From martin.senger at gmail.com Tue Jul 1 14:41:49 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 1 Jul 2008 11:41:49 -0700
Subject: [MOBY-dev] data by reference - a request for comments
Message-ID: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
Hi all,
Yesterday, Mark, Eddie and I, we spent some time to evaluate what was
proposed during our last meeting about sending data by references. Here are
some thoughts that may crystalize into a request fro comments.
*What is the purpose of sending data by reference
*
Well, the first purpose (A) is obvious: we want to be able to deliver huge
data from a service. So the service returns only a reference instead of the
real data and the client can fetch the real data using some memory-friendly
protocol (usualy a simple HTTP/GET).
But it appeeared that it was not the only purpose. The second purpose (B) is
to be able to send around already existing references (such as URLs of the
EMBL or NCBI records). The existence of this B purpose makes the problem a
bit harder but it is a valid puspose. So the first question is:
*Do you agree that we pursue both purposes in this requests?
*
*The machinery
*
a) A service claims (in the registration time) that it can provide data by
reference.
b) A client asks for getting back references by including "acceptRefs"
attribute in mobyData tag. The attribute lists one or more protocol names
that the client can accept.
c) A service *can* obey such request and send one or more *primitive
data*as references (the focus on primitive type is new, originaly we
thought
about allowing references on any level, but now, mainly becuse of the
purpose B we do not propose it anymore). It can use any of the protocols
mentioned in the client's "acceptRefs" attribute. It can send references
only if at least one protocol matches.
*How does a client knows what protocols a service supports?*
This is a fundamental question that goes closely with "use existing
standards rather than inventing your own". An ideal solution is perhaps
this: A service returns not a reference to data itself but a reference to a
WSDL document that contains all supported protocols, including the endpoints
for this particular data. It is a nice idea but it breaks the purpose B - we
cannot use existing references without wrapping them first in a WSDL
document. The WSDL is strong because it gives us actually an *interface* how
to get data, but it is weak because the references cannot be used as *
indexes* (e.g. for further caching). Also, it does not solve the client
side: the "acceptRefs" attribute still needs to use a list of protocol names
(and not a WSD document because clients cannot make WSDL documents visible
to the world).
After going there and back, we concluded (and it is now our proposal for the
request of comments) that the service returns a reference to data, and
clients can deduce what protocol to us by looking at the protocol part of
the returned URL. We are aware that this is fine for usual protocol, such as
HTTP and FTP, but it cannot serve data, for example, by a SOAP. But, as
Eddie pointed out, if somebody wants SOAP for data, she can return data
directly in the Moby message.
*The remaining questions
*
Dmitry suggested to use WSRF. We think that he meant something else: It
could be used instead of the whole Moby message - but that is not what we
are looking for. We are looking for replacing just data part by references,
and we want still to keep the original Moby message as it is used now. So we
have concluded: no WSRF.
*How can a client tells a service that she is sending a reference instead of
data?* This could be useful for chaining services. We have not talked about
it. Ideas welcome.
The machinery described above may not allow to find, in advance, what
protocols a service is able to provide. It depends on what a service can
register into a moby central registry. It can be just a boolen flag ("I can
provid references"), or a list of supported protocols ("names"), or actually
nothing. The latest option has an advantage that no change in the registry
is needed. *Can we live with this simple option?*
I am not sure if I covered all, but better to send ti now and wait for your
comments.
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From jmfernandez at cnio.es Tue Jul 1 15:04:40 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Tue, 01 Jul 2008 21:04:40 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
Message-ID: <486A7FC8.20201@cnio.es>
Hi everybody,
I have added inline some personal comments about the RFC:
Martin Senger wrote:
> Hi all,
>
> Yesterday, Mark, Eddie and I, we spent some time to evaluate what was
> proposed during our last meeting about sending data by references. Here are
> some thoughts that may crystalize into a request fro comments.
>
> *What is the purpose of sending data by reference
> *
> Well, the first purpose (A) is obvious: we want to be able to deliver huge
> data from a service. So the service returns only a reference instead of the
> real data and the client can fetch the real data using some memory-friendly
> protocol (usualy a simple HTTP/GET).
>
> But it appeeared that it was not the only purpose. The second purpose (B) is
> to be able to send around already existing references (such as URLs of the
> EMBL or NCBI records). The existence of this B purpose makes the problem a
> bit harder but it is a valid puspose. So the first question is:
>
> *Do you agree that we pursue both purposes in this requests?
> *
> *The machinery
> *
> a) A service claims (in the registration time) that it can provide data by
> reference.
>
It can be claimed from the RDF of the service as a predicate, including the
understood reference protocols (which should be described elsewhere in an RDF
ontology).
> b) A client asks for getting back references by including "acceptRefs"
> attribute in mobyData tag. The attribute lists one or more protocol names
> that the client can accept.
>
I suggest using serviceNotes child element (called for instance acceptRefs
:-), telling what it can understand) instead of adding a new attribute to
mobyData, in both ways of the MOBY dialog. The client uses the acceptRefs
element in the message sent to the service to tell what it can understand, and
the service answers using the acceptRefs explaining what it have done. The
service must answer a subset of the understood reference protocols to the
client in order to maintain the contract.
If the service cannot fulfill the request where client does not understand the
protocols' the service is able to handle (because it refuses to send too much
information inline), it can fail using a new exception code related to message
size.
If the service uses references which are not understood by the client, then
the client can mark the message or the service as corrupted or offensive.
> c) A service *can* obey such request and send one or more *primitive
> data*as references (the focus on primitive type is new, originaly we
> thought
> about allowing references on any level, but now, mainly becuse of the
> purpose B we do not propose it anymore). It can use any of the protocols
> mentioned in the client's "acceptRefs" attribute. It can send references
> only if at least one protocol matches.
>
> *How does a client knows what protocols a service supports?*
>
> This is a fundamental question that goes closely with "use existing
> standards rather than inventing your own". An ideal solution is perhaps
> this: A service returns not a reference to data itself but a reference to a
> WSDL document that contains all supported protocols, including the endpoints
> for this particular data. It is a nice idea but it breaks the purpose B - we
> cannot use existing references without wrapping them first in a WSDL
> document. The WSDL is strong because it gives us actually an *interface* how
> to get data, but it is weak because the references cannot be used as *
> indexes* (e.g. for further caching). Also, it does not solve the client
> side: the "acceptRefs" attribute still needs to use a list of protocol names
> (and not a WSD document because clients cannot make WSDL documents visible
> to the world).
>
A passive, static way is publishing the service capabilities about references
in the RDF. An active, dynamic way is creating a new service port (like we did
with asynchronous services), which provides live metadata information about
the service. I could even be linked/related to LSID metadata servers, but it
is still outside my knowledge.
> After going there and back, we concluded (and it is now our proposal for the
> request of comments) that the service returns a reference to data, and
> clients can deduce what protocol to us by looking at the protocol part of
> the returned URL. We are aware that this is fine for usual protocol, such as
> HTTP and FTP, but it cannot serve data, for example, by a SOAP. But, as
> Eddie pointed out, if somebody wants SOAP for data, she can return data
> directly in the Moby message.
>
> *The remaining questions
> *
> Dmitry suggested to use WSRF. We think that he meant something else: It
> could be used instead of the whole Moby message - but that is not what we
> are looking for. We are looking for replacing just data part by references,
> and we want still to keep the original Moby message as it is used now. So we
> have concluded: no WSRF.
>
> *How can a client tells a service that she is sending a reference instead of
> data?* This could be useful for chaining services. We have not talked about
> it. Ideas welcome.
>
> The machinery described above may not allow to find, in advance, what
> protocols a service is able to provide. It depends on what a service can
> register into a moby central registry. It can be just a boolen flag ("I can
> provid references"), or a list of supported protocols ("names"), or actually
> nothing. The latest option has an advantage that no change in the registry
> is needed. *Can we live with this simple option?*
>
> I am not sure if I covered all, but better to send ti now and wait for your
> comments.
>
> Cheers,
> Martin
>
Best Regards,
Jos? Mar?a
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.
From martin.senger at gmail.com Wed Jul 2 01:22:30 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 1 Jul 2008 22:22:30 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486A7FC8.20201@cnio.es>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
<486A7FC8.20201@cnio.es>
Message-ID: <4d93f07c0807012222i1ae4f97ai152796660a0bc46a@mail.gmail.com>
> It can be claimed from the RDF of the service as a predicate, including the
> understood reference protocols (which should be described elsewhere in an
> RDF ontology).
I do not like using the RDF for mandatory features. The RDF, at least in the
current moby, is optional and should not carry anything that is significant
for the service and its behaviour. The data by reference should be part of
the main BioMoby API (or whatever we call it).
> I suggest using serviceNotes child element
I do not have a strong opinion where in the message this could be - I have
just used the tag name and location that was suggested in our February
meeting. The ServiceNotes, however, seems a bit a strange place when used by
the clients.
> and the service answers using the acceptRefs explaining what it have done.
Why? Why is not enough the reference itself? It is a URL and the URL has a
wel-defined protocol. And what if a service uses a different protocol for
different parts of the message (remember the purpose B)? No, the
ServiceNotes is not a suitable place for it.
> The service must answer a subset of the understood reference protocols to
> the client in order to maintain the contract.
And what about if the service does not want to return references, at all?
That should be an error, as well? No, I don't think so. A client just
suggests that she is able to accept references but there is no guarantee
that she can get it (many services will not have any provision for
references).
> A passive, static way is publishing the service capabilities about
> references in the RDF.
It can be there, why not, - but not as the main way.
> An active, dynamic way is creating a new service port (like we did with
> asynchronous services)
Do you mean a new method name? But for what? Not sure I understand how this
new method may help here (and not breaking the purpose B, for example).
Just my opinions.
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From dmitry.repchevski at bsc.es Wed Jul 2 06:26:52 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Wed, 02 Jul 2008 12:26:52 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
Message-ID: <486B57EC.7090302@bsc.es>
Hello!
Here are my comments. Please note that all they are in mode IMHO,
because I understand the complicity of the problem and just looking for
possible solution.
> [A] Well, the first purpose (A) is obvious: we want to be able to deliver huge data from a service.
It depends.
If we are going to stay with SOAP (and especially move to doc/lit) the
right way is to provide streaming through SOAP Attachments (SwA) or even
better using MTOM.
This way we have to introduce a special Moby Primitive Type something
called "BinaryAttachment" which will be bind to [type="base64Binary"]
This type will be replaced with:
...
--MIME_boundary
Content Type:image/gif
Content-Transfer-Encoding:binary
Content-ID:
... (file content)
--MIME_boudary
In case of using MTOM:
...
------=_Part_1_4558657.1118953559446
Content-Type: image/jpeg
Content-ID:
Obviously JAVA supports both, but considering Perl, I am not sure which
is better
(I am very pessimistic in SOAP Perl capabilities... :-( )
> The second purpose (B) is to be able to send around already existing references (such as URLs of the EMBL or NCBI records).
Here is the question where ever we continue to invent standards or
trying to use SOAP capabilities...
Finally we have WS-Addressing (and even have it implemented in Perl :-) ).
More we have WSRF that is exactly for this purpose - reference for resource.
> Dmitry suggested to use WSRF. We think that he meant something else: It
> could be used instead of the whole Moby message - but that is not what we
> are looking for.
I am afraid you didn't understand me well... WSRF doesn't mean we have
to send ALL the message.
It is only the way to reference a resource. Even in moby-asynchrony we
return the whole message if needn't be so.
We can perfectly return a reference to ANY part of the original request.
It is up to the service to locate the asked resource and send it back to
the WSRF request.
We shouldn't confuse "A" and "B" because they are different.
"A" is to stream the data which in many case means "give me a huge file".
"B" is to keep the reference to it (something temporal like a
calculation that produces a lot of data and we don't want to pass it
around).
> We are looking for replacing just data part by references,
> and we want still to keep the original Moby message as it is used now. So we
> have concluded: no WSRF.
Don't understand it... How you suppose to replace some object in Moby
message by refernce without changing it?
> *How can a client tells a service that she is sending a reference instead of
> data?* This could be useful for chaining services. We have not talked about
> it. Ideas welcome.
There is no way to specify that it can be send both by value and by
reference and the same type...
I suggest - the service expecting the usual "value", but check if there
is WSRF resource sent in header.
If it is it just calls the service (or in case it is the same server,
just locate it locally) to get the object (not all Moby message, but
just serialized Moby object, like AminoacidSequence) and use it instead
the EMPTY one in MobyMessage.
Something like - We expect two AminoacidSequence(s) -
articleName="first", articleName="second"
In header we put WS Addressing (WSRF) resource to "server1/second"
The service check headers for resources and call the server1 to obtain
"second" Aminoacid.
This can be useful to make a workflow too... because we can call the
server1 to get AminoacidSequence2, obtain reference to it AND send it to
the server 2, so actually server2 will get the reference to it...
In case server1 and the server2 are the same (this is the main idea of
[B]?) we do not need to pass around and get it directly... in case of
server1 != server2 we have a ... Moby "GRID" :D
> I do not like using the RDF for mandatory features. The RDF, at least in the
> current moby, is optional and should not carry anything that is significant
> for the service and its behaviour.
Agree. IMHO I would use WSDL capability to describe it all, but having
so many types of services this could be very hard...
IMO the main problem is that BioMoby is using SOAP, but use nothing of
it trying to replace all W3C and WS-I specification by it's own solution.
So the question is why do we use SOAP?
I see (in a long term) two possibilities:
1. Forget about SOAP and do the REST (which is a fancy name to say we
are going to send what ever we want without any standard)
2. Use SOAP and Web Services architecture.
Best regards,
Dmitry
From markw at illuminae.com Wed Jul 2 10:22:13 2008
From: markw at illuminae.com (Mark)
Date: Wed, 02 Jul 2008 07:22:13 -0700
Subject: [MOBY-dev] Interesting...
Message-ID:
http://www.biomedcentral.com/1471-2105/9/295/abstract
I wonder how they got document-literal stuff working in Perl??
M
From markw at illuminae.com Wed Jul 2 10:11:29 2008
From: markw at illuminae.com (Mark)
Date: Wed, 02 Jul 2008 07:11:29 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486B57EC.7090302@bsc.es>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
<486B57EC.7090302@bsc.es>
Message-ID:
>> The second purpose (B) is to be able to send around already existing
>> references (such as URLs of the EMBL or NCBI records).
> Here is the question where ever we continue to invent standards or
> trying to use SOAP capabilities...
> Finally we have WS-Addressing (and even have it implemented in Perl :-)
> ).
> More we have WSRF that is exactly for this purpose - reference for
> resource.
Hi Dmitry!
Could you send an example of a Moby message that has multiple components
(e.g., the image and annotation component of SimpleAnnotatedJpegImage)
where the content of the data-carrying primitives are references using the
WS-Addressing standard? that way, we'll all know without any ambiguity
what it is you are suggesting.
Thanks!
Mark
From dmitry.repchevski at bsc.es Wed Jul 2 13:56:51 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Wed, 02 Jul 2008 19:56:51 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
Message-ID: <486BC163.9050007@bsc.es>
Hello Mark,
My suggestion (not a final specification :-)) is something like:
******************************
http://myserver.com/MyService?asyncId=ID
ID
mobyws:result_queryId00/image/jpeg/content
description
***********************
The idea is to include the WSRF reference into the header:
***********************
http://myserver.com/MyService?asyncId=ID
ID
mobyws:result_queryId00/image/jpeg/content
***********************
The service can access the header looking for resources and then to call
WSRF service specified to obtain them.
Of course we must update our WSRF stack to return a particular moby
object instead of all mobyData. here I "updated" the like to
"mobyws:result_queryId00/image/jpeg/content"
Note that I used a simplest possible way to produce an... uri (?) to it
- using "articleNames". I commented the "content" in the body to
demonstrate the idea better - it is up to service implementation to call
a wsrf and obtain the image... This way we do not need to change "moby"
message.
Cheers,
Dmitry
From gordonp at ucalgary.ca Wed Jul 2 13:57:40 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Wed, 02 Jul 2008 11:57:40 -0600
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807012222i1ae4f97ai152796660a0bc46a@mail.gmail.com>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com> <486A7FC8.20201@cnio.es>
<4d93f07c0807012222i1ae4f97ai152796660a0bc46a@mail.gmail.com>
Message-ID: <486BC194.90506@ucalgary.ca>
Hi Martin,
I agree with most of your post, but I will be pedantic and point out the
trap you, and previously Tom Oinn, fell into in this discussion:
> Why? Why is not enough the reference itself? It is a URL and the URL has a
> wel-defined protocol. ...
URL does not in fact have a well-defined protocol. URL is simply a way
to express "protocol:protocol-specific-stuff"
I'll assume you mean http and maybe ftp protocols. It's important to be
specific, because you probably don't want to include mailto:, gopher: or
prospero: for example.
Paul
From markw at illuminae.com Wed Jul 2 14:21:57 2008
From: markw at illuminae.com (Mark)
Date: Wed, 02 Jul 2008 11:21:57 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486BC163.9050007@bsc.es>
References: <486BC163.9050007@bsc.es>
Message-ID:
Do you suggest that there is no indication, in the MOBY block, that the
message contains data-by-reference? i.e. should our tools simply see that
the primitive is empty, and then look into the ws header for the
possibility of a ws-ref, and then if it isn't found, adopt the other
interpretation of an empty block (i.e. that the data is known to be NULL)
?
M
On Wed, 02 Jul 2008 10:56:51 -0700, Dmitry Repchevsky
wrote:
> Hello Mark,
>
> My suggestion (not a final specification :-)) is something like:
>
> ******************************
>
>
>
>
> xmlns:wsa="http://www.w3.org/2005/08/addressing">
> http://myserver.com/MyService?asyncId=ID
>
>
> ID
>
>
>
> mobyws:result_queryId00/image/jpeg/content
>
>
>
>
>
>
>
>
>
>
>
> articleName="Description">description
>
>
>
>
>
>
>
>
> ***********************
>
> The idea is to include the WSRF reference into the header:
>
> ***********************
> xmlns:wsa="http://www.w3.org/2005/08/addressing">
> http://myserver.com/MyService?asyncId=ID
>
> xmlns:mobyws="http://biomoby.org/">ID
>
>
> mobyws:result_queryId00/image/jpeg/content
>
>
> ***********************
>
> The service can access the header looking for resources and then to call
> WSRF service specified to obtain them.
> Of course we must update our WSRF stack to return a particular moby
> object instead of all mobyData. here I "updated" the like to
> "mobyws:result_queryId00/image/jpeg/content"
> Note that I used a simplest possible way to produce an... uri (?) to it
> - using "articleNames". I commented the "content" in the body to
> demonstrate the idea better - it is up to service implementation to call
> a wsrf and obtain the image... This way we do not need to change "moby"
> message.
>
> Cheers,
>
> Dmitry
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
--
--
Mark Wilkinson
Heart + Lung Research Institute
Centre for High Throughput Biology
UBC
From dmitry.repchevski at bsc.es Wed Jul 2 18:15:13 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 03 Jul 2008 00:15:13 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
Message-ID: <486BFDF1.1070001@bsc.es>
Hello,
No I think that the service (tool?) should look into the header first
and if there are some references, ignore corresponding element from the
body. As I said this is only an idea. This way we do not change the
message itself and do can implement it on the base of existing wsrf code...
As of the wsrf response for
//"mobyws:result_queryId00/image/jpeg/content"// the service will return
the standard MOBY message but only with elements found in the path
(MOBY/mobyContent/mobyData/Simple/SimpleAnnotatedJPEGImage/String
Best regards,
Dmitry
From edward.kawas at gmail.com Wed Jul 2 18:33:07 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 2 Jul 2008 15:33:07 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486BFDF1.1070001@bsc.es>
References: <486BFDF1.1070001@bsc.es>
Message-ID: <486c0227.07ec720a.5dc1.ffffd76a@mx.google.com>
With all this talk about WSRF, SOAP, etc, what would happen with cgi (i.e.
non moby/moby-async) services?
I think that the solution needs to be able to encompass all categories of
services that moby claims to support and not just [JAVA] SOAP services.
Just a few of my thoughts,
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Dmitry Repchevsky
Sent: July-02-08 3:15 PM
To: moby-dev at lists.open-bio.org
Subject: Re: [MOBY-dev] data by reference - a request for comments
Hello,
No I think that the service (tool?) should look into the header first
and if there are some references, ignore corresponding element from the
body. As I said this is only an idea. This way we do not change the
message itself and do can implement it on the base of existing wsrf code...
As of the wsrf response for
//"mobyws:result_queryId00/image/jpeg/content"// the service will return
the standard MOBY message but only with elements found in the path
(MOBY/mobyContent/mobyData/Simple/SimpleAnnotatedJPEGImage/String
Best regards,
Dmitry
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From martin.senger at gmail.com Thu Jul 3 00:11:26 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Wed, 2 Jul 2008 21:11:26 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486BC194.90506@ucalgary.ca>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
<486A7FC8.20201@cnio.es>
<4d93f07c0807012222i1ae4f97ai152796660a0bc46a@mail.gmail.com>
<486BC194.90506@ucalgary.ca>
Message-ID: <4d93f07c0807022111u4f65fe7ap851a997a23e230a8@mail.gmail.com>
Paul,
URL does not in fact have a well-defined protocol. URL is simply a way to
> express "protocol:protocol-specific-stuff"
Point taken. The URL has a well-defined *place* where I can find a protocol,
but it does not define what protocol can sit on that place.
> I'll assume you mean http and maybe ftp protocols. It's important to be
> specific, because you probably don't want to include mailto:, gopher: or
> prospero: for example.
>
Well, I did not mean to be too specific. All the protocols you mentioned
would be possible (but some of them may hardly find a match between a
service an a client).
All,
I am not against standards, actually standards was what I was hired to
promote at EBI, but my experiences are that sometimes standards simply are
an obstacle. Especially for BioMoby - that is anyway a non-standard message
protocol. We already burned ourselves with using WSRF for asynchronous
services (we have now problems how to make asynchronous services using
REST). I am afraid that using WSRF will rather move service provider from
Moby world than closer to it (I am talking from the point of view of the
service providers and the tools developers, not from the perspective of Mark
who has to face the funding agencies).
The fact is that we want to use REST (in the near future) and not only SOAP,
we want to use Perl where SOAP is not that well supported, and we want the
solution for data by reference to be simple.
Therefore, I do not wish to be eaten (again) by the WSRF - unless my boss
(whoever it is or will be) forces me to do so, of course.
BTW, I like very much the Dmitry's definition of the REST ("a fancy name to
say we are going to send what ever we want without any standard"). I agree
with him - except that we are sending Moby message - which is not standard,
it is faulty, but it has worked relatively fine so far.
For being able to continue this discussion (about data by reference), I
suggest that we try to answer first whether we want to use only SOAP or
whether we are really heading to use REST, as well. My vote for this
question is: yes, we will want to have the REST services, as well.
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From dmitry.repchevski at bsc.es Thu Jul 3 04:46:40 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 03 Jul 2008 10:46:40 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486c0227.07ec720a.5dc1.ffffd76a@mx.google.com>
References: <486c0227.07ec720a.5dc1.ffffd76a@mx.google.com>
Message-ID: <486C91F0.4000100@bsc.es>
>
> With all this talk about WSRF, SOAP, etc, what would happen with cgi (i.e.
> non moby/moby-async) services?
Interesting question... How do we implement all possible functionality
using all possible solutions... :-)
Here is my point of view:
1. "moby" service
It can work with "wsrf" pass by reference... it's up to the toolkit
to look into the header
Attachment (SwA). Just put your attachment outside the soap message
and find it somehow from the toolkit
2 "moby-async" service
The same as "moby", but also can return a reference (through wsrf) to
a particular moby object
3. "doc-literal" service
The same as "moby", but attachment would be MTOM (backward compatible
with SwA).
4. "doc-literal-async"
Also see no problem.
5. "soap", "cgi", "wsdl", "post" - what's this??? (sorry for ingnorance)
Best regards,
Dmitry
From machiel.jansen at sara.nl Thu Jul 3 05:13:14 2008
From: machiel.jansen at sara.nl (Machiel Jansen)
Date: Thu, 3 Jul 2008 11:13:14 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486C91F0.4000100@bsc.es>
References: <486c0227.07ec720a.5dc1.ffffd76a@mx.google.com>
<486C91F0.4000100@bsc.es>
Message-ID:
For all I know WSRF is already been surpassed by WS-RT. http://www.ibm.com/developerworks/grid/library/gr-wsrfwsrt/index.html?ca=drs-
There are quite a number of people who are disenchanted by WSRF. See http://blog.harbulot.com/post/2006/11/22/Experiences-with-WSRF for one example.
Machiel
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org [mailto:moby-dev-
> bounces at lists.open-bio.org] On Behalf Of Dmitry Repchevsky
> Sent: donderdag 3 juli 2008 10:47
> To: moby-dev at lists.open-bio.org
> Subject: Re: [MOBY-dev] data by reference - a request for comments
>
> >
> > With all this talk about WSRF, SOAP, etc, what would happen with cgi
> (i.e.
> > non moby/moby-async) services?
> Interesting question... How do we implement all possible functionality
> using all possible solutions... :-)
>
> Here is my point of view:
>
> 1. "moby" service
> It can work with "wsrf" pass by reference... it's up to the toolkit
> to look into the header
> Attachment (SwA). Just put your attachment outside the soap message
> and find it somehow from the toolkit
>
> 2 "moby-async" service
> The same as "moby", but also can return a reference (through wsrf)
> to
> a particular moby object
>
> 3. "doc-literal" service
> The same as "moby", but attachment would be MTOM (backward
> compatible
> with SwA).
>
> 4. "doc-literal-async"
> Also see no problem.
>
>
> 5. "soap", "cgi", "wsdl", "post" - what's this??? (sorry for ingnorance)
>
> Best regards,
>
> Dmitry
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
From dmitry.repchevski at bsc.es Thu Jul 3 05:20:32 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 03 Jul 2008 11:20:32 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807022111u4f65fe7ap851a997a23e230a8@mail.gmail.com>
References: <4d93f07c0807022111u4f65fe7ap851a997a23e230a8@mail.gmail.com>
Message-ID: <486C99E0.4040402@bsc.es>
Hello Martin,
So as far as I understand, the final battle is about Rest/Soap... So
let's use CORBA!!! (joke)
I am not the one that like all the standards (especially the
latest_and_greatest ones), but as a developer I recognize that we can
not compete against the whole industry. My usual position is to use
those standards that have a strong support in a Java world (of course I
am Java biased). So I see what we have from w3c, an then to JCP... I
don't want to implement them by myself... I am lazy...
With WSRF I had no other way, but implement a part of it. Here we have a
problem, because in distributed computations (and Moby is a kind of it)
Grid is very popular and so WSRF, but somehow Sun guys ignore it...
I understand SOAP and how it works... We can do all the things we need
just USING it.
Even current Moby message format can fit into it with XML Schema. My
moby implementation library doesn't parse moby message, but rather let
the standard Java library do it.
I am afraid we can not mix SOAP-WSRF approach with REST... Just because
the REST is not a technology, neither a protocol.
It's just a concept (and... IT IS STATELESS), so you can define your
protocol, but you will need to reinvent all the protocols... when a
resource should be destroyed? It reminds me the year 2000 when having
CORBA that worked just fine, someone said "It is complicated, let's
invent something simple"... Now we have SOAP with 4 different encodings
+ some 20 specifications + WSDL +... And all they do the same CORBA did
10 years ago...
Now people don't want SOAP because it is complicated so let's start
REST... OK, but I'd like to see the specification with
asynchrony/streaming based on REST concept.
What is why I love Java - There is no any Java specification without a
working implementation.
So, finally IMHO if we want to move out of SOAP we at least need to know
where we are moving...
Cheers,
Dmitry
From dmitry.repchevski at bsc.es Thu Jul 3 05:57:59 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 03 Jul 2008 11:57:59 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
Message-ID: <486CA2A7.4070702@bsc.es>
Hello,
> For all I know WSRF is already been surpassed by WS-RT. http://www.ibm.com/developerworks/grid/library/gr-wsrfwsrt/index.html?ca=drs-
> There are quite a number of people who are disenchanted by WSRF. See http://blog.harbulot.com/post/2006/11/22/Experiences-with-WSRF for one example.
>
> Machiel
This is what I am talking about... Martin has a reason we have had
enough problems with WSRF... now we have a "better" one (WSRT)...
But (!) There is an implementation? It looks like that this is a perfect
technology that exists in a paper...
Should we implement all the standards just to be able to use it? If IBM
claims to have a new/cool standard (by the way "standard" - who is
behind it?) but why can't I find (free) implementation of it?
It uses WS-Transfer has more sense to me because it can describe a way
to represent a moby object in moby message tree (for wsrf). and it is at
least w3s standard.
Claiming something to be a standard is always a bold claim...
Dmitry
From jflo at imim.es Tue Jul 8 07:42:04 2008
From: jflo at imim.es (Judith Flo)
Date: Tue, 08 Jul 2008 13:42:04 +0200
Subject: [MOBY-dev] strange behaviour
Message-ID: <4873528C.7070000@imim.es>
Hi,
Yesterday I changed one datatype (WeightedObject) to add another String
(Description). I unregistered the previous one and registered the new
one, same name, diferent structure.
Now the problem: when I try to compile the service, i get an error
telling me that the method that should set, this new attribute doesn't
exists. It is a very strange problem I've never seen before... The
curious is that Eclipse is not complaining and when I type object.set it
shows me as a possible operation to do, to set this atribute, but when I
try to compile it, just get:
[javac] Compiling 1 source file to
/home/judith/workspace3/services/WebContent/WEB-INF/classes
[javac]
/home/judith/workspace3/services/src/es/imim/cgl/runBlastXPImpl.java:76:
cannot find symbol
[javac] symbol : method set_Description(java.lang.String)
[javac] location: class org.biomoby.shared.datatypes.WeightedObject
[javac] w.set_Description("asdf");
[javac] ^
[javac] 1 error
Does someone know what's happening?
Of course I have regenerated the biomoby-datatypes.jar and now I check
that the new class exists and it contains this new atribute.
j
From jflo at imim.es Tue Jul 8 07:58:07 2008
From: jflo at imim.es (Judith Flo)
Date: Tue, 08 Jul 2008 13:58:07 +0200
Subject: [MOBY-dev] strange behaviour part 2
Message-ID: <4873564F.4020700@imim.es>
I also got this error on server side when trying to run some service
from dashboard:
AN ERROR OCCURED DURING THE SERVICE EXECUTION: Class
'org.biomoby.shared.datatypes.MapDataTypes' was not found.
It may indicate that you have not generated all Biomoby data types from
a Biomoby registry.
See http://www.biomoby.org/moby-live/Java/docs/Moses.html for details.
If you are a jMoby developer just type: ./build-dev.sh moses-datatypes.
Or perhaps, they just need to be compiled: ./build-dev.sh moses-compile.
From martin.senger at gmail.com Tue Jul 8 09:07:01 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 8 Jul 2008 08:07:01 -0500
Subject: [MOBY-dev] strange behaviour
In-Reply-To: <4873528C.7070000@imim.es>
References: <4873528C.7070000@imim.es>
Message-ID: <4d93f07c0807080607i402e659cy22532236854192d@mail.gmail.com>
> Yesterday I changed one datatype (WeightedObject) to add another String
> (Description). I unregistered the previous one and registered the new one,
> same name, diferent structure.
When changing an existing data type, the local cache should be updated. Or
newly created (reloading).
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From martin.senger at gmail.com Tue Jul 8 09:10:24 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 8 Jul 2008 08:10:24 -0500
Subject: [MOBY-dev] strange behaviour part 2
In-Reply-To: <4873564F.4020700@imim.es>
References: <4873564F.4020700@imim.es>
Message-ID: <4d93f07c0807080610p75acfcdaofc707d78abc1cb5a@mail.gmail.com>
> AN ERROR OCCURED DURING THE SERVICE EXECUTION: Class
> 'org.biomoby.shared.datatypes.MapDataTypes' was not found.
>
Look into your Tomcat's WEB-INF's lib directory and check what jar files you
have there, and if there are up-to-date.
> If you are a jMoby developer just type: ./build-dev.sh moses-datatypes.
>
Be aware that this message is a bit old (I have to update it in the code).
But you already know that you are using rather "ant moses-datatypes".
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From jflo at imim.es Tue Jul 8 10:51:14 2008
From: jflo at imim.es (Judith Flo)
Date: Tue, 08 Jul 2008 16:51:14 +0200
Subject: [MOBY-dev] strange behaviour part 2
In-Reply-To: <4d93f07c0807080610p75acfcdaofc707d78abc1cb5a@mail.gmail.com>
References: <4873564F.4020700@imim.es>
<4d93f07c0807080610p75acfcdaofc707d78abc1cb5a@mail.gmail.com>
Message-ID: <48737EE2.1010804@imim.es>
I've checked the lasts versions that are running (both in my machine and
the server) and the biomoby-datatypes.jar is the same (the last that I
generated with the object changed).
I did reload the cache before, I repeated now and it keeps complaining
about this new atribute/method.
I'll try to remove cache... maybe it'll help...
Any other idea?
j
Martin Senger wrote:
>> AN ERROR OCCURED DURING THE SERVICE EXECUTION: Class
>> 'org.biomoby.shared.datatypes.MapDataTypes' was not found.
>>
>>
>
> Look into your Tomcat's WEB-INF's lib directory and check what jar files you
> have there, and if there are up-to-date.
>
>
>
>> If you are a jMoby developer just type: ./build-dev.sh moses-datatypes.
>>
>>
>
> Be aware that this message is a bit old (I have to update it in the code).
> But you already know that you are using rather "ant moses-datatypes".
>
> Martin
>
>
From martin.senger at gmail.com Tue Jul 8 11:27:27 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 8 Jul 2008 10:27:27 -0500
Subject: [MOBY-dev] strange behaviour part 2
In-Reply-To: <48737EE2.1010804@imim.es>
References: <4873564F.4020700@imim.es>
<4d93f07c0807080610p75acfcdaofc707d78abc1cb5a@mail.gmail.com>
<48737EE2.1010804@imim.es>
Message-ID: <4d93f07c0807080827n1b63ef3yd4c8cfdc420983c6@mail.gmail.com>
> Any other idea?
Also restart yout Tomcat. Or, try your service "locally" (either from
Dashboard or from a command-line script) - if this works then the probelm is
in Tomcat.
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Mon Jul 14 02:52:10 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Mon, 14 Jul 2008 08:52:10 +0200
Subject: [MOBY-dev] Problem with MobyNamespace.getNamespace
Message-ID: <487AF79A.30301@mpiz-koeln.mpg.de>
Hi,
I'm using the method MobyNamespace.getNamespace(String ns) in the jmoby
api to retrieve the MobyNamespace for a given string. Unfortunately this
method is not working anymore correctly. E.g. if i use PMID as input
(which is a namespace) i get null back from this method.
I tested it with PDB, AFAWE_ID, PMID and UniProt, which all are namespaces.
The java code I'm using is some weeks old and it worked as far as i know
for a long long time - why not now ?
If i call http://biomoby.org/RESSOURCES/MOBY-S/Namespaces i get the rdf
in which I do find e.g. PMID... but still the method above returns null.
any ideas ?
thx
andreas
From gordonp at ucalgary.ca Mon Jul 14 09:57:02 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Mon, 14 Jul 2008 07:57:02 -0600
Subject: [MOBY-dev] Problem with MobyNamespace.getNamespace
In-Reply-To: <487AF79A.30301@mpiz-koeln.mpg.de>
References: <487AF79A.30301@mpiz-koeln.mpg.de>
Message-ID: <487B5B2E.4030104@ucalgary.ca>
Hi Andreas,
I'll check it out and get back to you...
Andreas Groscurth wrote:
> Hi,
>
> I'm using the method MobyNamespace.getNamespace(String ns) in the
> jmoby api to retrieve the MobyNamespace for a given string.
> Unfortunately this method is not working anymore correctly. E.g. if i
> use PMID as input (which is a namespace) i get null back from this
> method.
>
> I tested it with PDB, AFAWE_ID, PMID and UniProt, which all are
> namespaces.
>
> The java code I'm using is some weeks old and it worked as far as i
> know for a long long time - why not now ?
>
> If i call http://biomoby.org/RESSOURCES/MOBY-S/Namespaces i get the
> rdf in which I do find e.g. PMID... but still the method above returns
> null.
>
> any ideas ?
>
> thx
> andreas
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
>
From groscurt at mpiz-koeln.mpg.de Tue Jul 15 07:56:17 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Tue, 15 Jul 2008 13:56:17 +0200
Subject: [MOBY-dev] findService and the hierarchy
Message-ID: <487C9061.3030203@mpiz-koeln.mpg.de>
Hi,
another question from my side...
I use the findService method in jmoby in CentralImpl. Due to the code
all calls end up in the method findService(MobyService service, String[]
keyword, boolean includeChildrenServiceType, boolean
includeParentDatatype).
I use the MobyDatatype Annotated_GO_Term which is a child of GO_Term. In
my opinion when searching for services consuming Annotated_GO_Term via
the findService method i should also include the service which consumes
a GO_Term datatype, as this is the parent datatype (per default
'includeParentDatatype' is true)....
what is wrong in this way to think ? because i do not find service with
datatype GO_Term ? and if the way i think is wrong - can you give me an
example how this parameter is used at all ?
Thanks
andreas
From gordonp at ucalgary.ca Wed Jul 16 16:43:02 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Wed, 16 Jul 2008 14:43:02 -0600
Subject: [MOBY-dev] Problem with MobyNamespace.getNamespace
In-Reply-To: <487AF79A.30301@mpiz-koeln.mpg.de>
References: <487AF79A.30301@mpiz-koeln.mpg.de>
Message-ID: <487E5D56.8050508@ucalgary.ca>
Hi Andreas,
I've updated MobyNamespace to use Eddie's new caching mechanism, and it
seems to work again. You'll ned to do a CVS update for the whole the
moby-live/Java/src/main directory. Please let me know if you have any
more trouble. I will be updating MobyDataType and MobyServiceType
similarly.
Regards,
Paul
Andreas Groscurth wrote:
> Hi,
>
> I'm using the method MobyNamespace.getNamespace(String ns) in the
> jmoby api to retrieve the MobyNamespace for a given string.
> Unfortunately this method is not working anymore correctly. E.g. if i
> use PMID as input (which is a namespace) i get null back from this
> method.
>
> I tested it with PDB, AFAWE_ID, PMID and UniProt, which all are
> namespaces.
>
> The java code I'm using is some weeks old and it worked as far as i
> know for a long long time - why not now ?
>
> If i call http://biomoby.org/RESSOURCES/MOBY-S/Namespaces i get the
> rdf in which I do find e.g. PMID... but still the method above returns
> null.
>
> any ideas ?
>
> thx
> andreas
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
>
From groscurt at mpiz-koeln.mpg.de Fri Jul 18 04:00:34 2008
From: groscurt at mpiz-koeln.mpg.de (groscurt)
Date: Fri, 18 Jul 2008 10:00:34 +0200
Subject: [MOBY-dev] Problem with MobyNamespace.getNamespace
In-Reply-To: <487E5D56.8050508@ucalgary.ca>
Message-ID:
Hi Paul
thanks for your effort, it does work now :)
Best
andreas
----------------original message-----------------
From: "Paul Gordon" gordonp at ucalgary.ca
To: "Core developer announcements" moby-dev at lists.open-bio.org
Date: Wed, 16 Jul 2008 14:43:02 -0600
-------------------------------------------------
> Hi Andreas,
>
> I've updated MobyNamespace to use Eddie's new caching mechanism, and it
> seems to work again. You'll ned to do a CVS update for the whole the
> moby-live/Java/src/main directory. Please let me know if you have any
> more trouble. I will be updating MobyDataType and MobyServiceType
> similarly.
>
> Regards,
>
> Paul
>
> Andreas Groscurth wrote:
>> Hi,
>>
>> I'm using the method MobyNamespace.getNamespace(String ns) in the
>> jmoby api to retrieve the MobyNamespace for a given string.
>> Unfortunately this method is not working anymore correctly. E.g. if i
>> use PMID as input (which is a namespace) i get null back from this
>> method.
>>
>> I tested it with PDB, AFAWE_ID, PMID and UniProt, which all are
>> namespaces.
>>
>> The java code I'm using is some weeks old and it worked as far as i
>> know for a long long time - why not now ?
>>
>> If i call http://biomoby.org/RESSOURCES/MOBY-S/Namespaces i get the
>> rdf in which I do find e.g. PMID... but still the method above returns
>> null.
>>
>> any ideas ?
>>
>> thx
>> andreas
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
From martin.senger at gmail.com Tue Jul 22 23:09:48 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 22 Jul 2008 20:09:48 -0700
Subject: [MOBY-dev] findService and the hierarchy
In-Reply-To: <487C9061.3030203@mpiz-koeln.mpg.de>
References: <487C9061.3030203@mpiz-koeln.mpg.de>
Message-ID: <4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
> In my opinion when searching for services consuming Annotated_GO_Term via
> the findService method i should also include the service which consumes a
> GO_Term datatype, as this is the parent datatype (per default
> 'includeParentDatatype' is true)....
>
I assume that by "I should also include" you actually meant "the registry
response should also include". Is my understanding correct?
If so, I am afraid that it is not the issue of the jMoby library but the
registry itself. Eddie or Mark can confirm or reject this hypothesis.
The BioMoby API allows to specify two XML tags when calling findService
method: the
X
and
X
Where X can be either 1 or 0.
The jMoby's findService() method sets these two tags according to the two
boolean parameters that the method can get:
boolean includeChildrenServiceTypes
and
boolean includeParentDataTypes
I have tried to call the findService, with the Annotated_GO_Term input,
using all combinations of these two tags (you can do it using the Moby
command-line client with the optioins -nc and/or -np). The results are
either no service or 713 services:
expandObjects expandServices # of found services
1 1 713
0 0 0
1 0 713
0 0 0
I have feeling that this is not a correct result. Could Eddie or Mark
perhaps comment on it?
[I vaguely remember that we had this conversation about these tags with Mark
years(!) ago - but I do not recall what was the conclusion. I only, and
again vaguely, remember to have this issue in my to-do list - which would
indicate that something could be done in the jMoby library.]
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Wed Jul 23 04:05:48 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 23 Jul 2008 10:05:48 +0200
Subject: [MOBY-dev] findService and the hierarchy
In-Reply-To: <4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
References: <487C9061.3030203@mpiz-koeln.mpg.de>
<4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
Message-ID: <4886E65C.5060205@mpiz-koeln.mpg.de>
Yes Martin you understood it - as always - perfectly correct :)
Thanks for the additional information
Martin Senger wrote:
>> In my opinion when searching for services consuming Annotated_GO_Term via
>> the findService method i should also include the service which consumes a
>> GO_Term datatype, as this is the parent datatype (per default
>> 'includeParentDatatype' is true)....
>>
>>
>
> I assume that by "I should also include" you actually meant "the registry
> response should also include". Is my understanding correct?
>
> If so, I am afraid that it is not the issue of the jMoby library but the
> registry itself. Eddie or Mark can confirm or reject this hypothesis.
>
> The BioMoby API allows to specify two XML tags when calling findService
> method: the
>
> X
> and
> X
>
> Where X can be either 1 or 0.
>
> The jMoby's findService() method sets these two tags according to the two
> boolean parameters that the method can get:
>
> boolean includeChildrenServiceTypes
> and
> boolean includeParentDataTypes
>
> I have tried to call the findService, with the Annotated_GO_Term input,
> using all combinations of these two tags (you can do it using the Moby
> command-line client with the optioins -nc and/or -np). The results are
> either no service or 713 services:
>
> expandObjects expandServices # of found services
> 1 1 713
> 0 0 0
> 1 0 713
> 0 0 0
>
> I have feeling that this is not a correct result. Could Eddie or Mark
> perhaps comment on it?
>
> [I vaguely remember that we had this conversation about these tags with Mark
> years(!) ago - but I do not recall what was the conclusion. I only, and
> again vaguely, remember to have this issue in my to-do list - which would
> indicate that something could be done in the jMoby library.]
>
> Cheers,
> Martin
>
>
From groscurt at mpiz-koeln.mpg.de Wed Jul 23 04:54:56 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 23 Jul 2008 10:54:56 +0200
Subject: [MOBY-dev] working with images
In-Reply-To:
References:
Message-ID: <4886F1E0.5020704@mpiz-koeln.mpg.de>
Hi,
i try to figure out how to transfer images via Web Service in moby....
E.g. i use the b64_encoded_jpeg. It has the children 'content' which
requires a String, so i need to somehow encode/decode the image file so
that it can be transmitted as a String and the client can read it as an
image (which is somehow done via MobyDataBytes)...
how to do that ?
thanks
Andreas
From Sebastien.Carrere at toulouse.inra.fr Wed Jul 23 05:10:04 2008
From: Sebastien.Carrere at toulouse.inra.fr (Sebastien Carrere)
Date: Wed, 23 Jul 2008 11:10:04 +0200
Subject: [MOBY-dev] working with images
In-Reply-To: <4886F1E0.5020704@mpiz-koeln.mpg.de>
References:
<4886F1E0.5020704@mpiz-koeln.mpg.de>
Message-ID: <4886F56C.4090100@toulouse.inra.fr>
Hi Andreas,
In my web services (written using PlayMoby , then Perl), I use the
module MIME::Base64
(http://search.cpan.org/~gaas/MIME-Base64-3.07/Base64.pm).
I just make a cat on my picture file, and encode the result in Base64,
then add it to my Moby String content.
Sebastien
Andreas Groscurth wrote:
> Hi,
>
> i try to figure out how to transfer images via Web Service in moby....
>
> E.g. i use the b64_encoded_jpeg. It has the children 'content' which
> requires a String, so i need to somehow encode/decode the image file
> so that it can be transmitted as a String and the client can read it
> as an image (which is somehow done via MobyDataBytes)...
>
> how to do that ?
>
> thanks
> Andreas
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Sebastien_Carrere.vcf
Type: text/x-vcard
Size: 387 bytes
Desc: not available
URL:
From martin.senger at gmail.com Wed Jul 23 05:22:19 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Wed, 23 Jul 2008 17:22:19 +0800
Subject: [MOBY-dev] working with images
In-Reply-To: <4886F1E0.5020704@mpiz-koeln.mpg.de>
References:
<4886F1E0.5020704@mpiz-koeln.mpg.de>
Message-ID: <4d93f07c0807230222l1586b9f1r37b0c9f05f326c38@mail.gmail.com>
> i try to figure out how to transfer images via Web Service in moby....
You can look for an example in the
org.biomoby.service.dashboard.renderers.Base64Image.java. This example does
opposite thing that you want, but it is still the same. I used there the
library coming with Axis (which is used by BioMoby anyway, so it is not an
additional dependency):
import org.apache.axis.encoding.Base64;
The example uses method decode() - and you need to use encode() (I guess).
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Wed Jul 23 05:36:25 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 23 Jul 2008 11:36:25 +0200
Subject: [MOBY-dev] working with images
In-Reply-To: <4886F56C.4090100@toulouse.inra.fr>
References: <4886F1E0.5020704@mpiz-koeln.mpg.de>
<4886F56C.4090100@toulouse.inra.fr>
Message-ID: <4886FB99.1080404@mpiz-koeln.mpg.de>
Hi,
thanks for that - i forgot to mention i'm looking for a java solution ;-)
but your hint of encode in Base64 made me google again and found the
apache.commons.codec and it works.
Thanks for the hint :)
Andreas
Sebastien Carrere wrote:
> Hi Andreas,
>
> In my web services (written using PlayMoby , then Perl), I use the
> module MIME::Base64
> (http://search.cpan.org/~gaas/MIME-Base64-3.07/Base64.pm).
>
> I just make a cat on my picture file, and encode the result in Base64,
> then add it to my Moby String content.
>
>
> Sebastien
>
>
>
>
>
> Andreas Groscurth wrote:
>> Hi,
>>
>> i try to figure out how to transfer images via Web Service in moby....
>>
>> E.g. i use the b64_encoded_jpeg. It has the children 'content' which
>> requires a String, so i need to somehow encode/decode the image file
>> so that it can be transmitted as a String and the client can read it
>> as an image (which is somehow done via MobyDataBytes)...
>>
>> how to do that ?
>>
>> thanks
>> Andreas
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
From martin.senger at gmail.com Wed Jul 23 05:35:32 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Wed, 23 Jul 2008 17:35:32 +0800
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
Message-ID: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
I have deployed (to Maven) the current jMoby as version 1.1.0.
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From edward.kawas at gmail.com Wed Jul 23 08:54:08 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 23 Jul 2008 05:54:08 -0700
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
Message-ID: <488729f9.20f8720a.0551.0626@mx.google.com>
Hi Martin,
Does this update include org.biomoby.service.dashboard.* (the dashboard
components)?
Thanks,
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
Sent: July-23-08 2:36 AM
To: Core developer announcements
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
I have deployed (to Maven) the current jMoby as version 1.1.0.
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From martin.senger at gmail.com Wed Jul 23 09:04:10 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Wed, 23 Jul 2008 21:04:10 +0800
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <488729f9.20f8720a.0551.0626@mx.google.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
<488729f9.20f8720a.0551.0626@mx.google.com>
Message-ID: <4d93f07c0807230604p664530ag2dda536268aaf8a1@mail.gmail.com>
> Does this update include org.biomoby.service.dashboard.* (the dashboard
> components)?
I have not changed anything regarding included packages - my intention was
simply to have the latest code available from maven (because we need it for
some projects). I am aware that you want to have some separation of packages
- for that we may make another release sooner or later.
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From jmfernandez at cnio.es Wed Jul 23 09:04:02 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Wed, 23 Jul 2008 15:04:02 +0200
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <488729f9.20f8720a.0551.0626@mx.google.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
<488729f9.20f8720a.0551.0626@mx.google.com>
Message-ID: <48872C42.3060008@cnio.es>
Hi Eddie and Martin,
I have realized that there are two sets of jMoby artifacts. The first one
(biomoby.org maven group) is the used by taverna-biomoby plugin, meanwhile the
second one (org.biomoby maven group) is the officially provided by you.
So, which are the differences between them? The first one is already in
version 1.2, while the second one has just reached version 1.1.0.
Best Regards,
Jos? Mar?a
Edward Kawas wrote:
> Hi Martin,
>
> Does this update include org.biomoby.service.dashboard.* (the dashboard
> components)?
>
> Thanks,
>
> Eddie
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
> Sent: July-23-08 2:36 AM
> To: Core developer announcements
> Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
>
> I have deployed (to Maven) the current jMoby as version 1.1.0.
>
> Cheers,
> Martin
>
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.
From edward.kawas at gmail.com Wed Jul 23 09:13:19 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 23 Jul 2008 06:13:19 -0700
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <48872C42.3060008@cnio.es>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com> <488729f9.20f8720a.0551.0626@mx.google.com>
<48872C42.3060008@cnio.es>
Message-ID: <48872e78.16538c0a.696c.1560@mx.google.com>
Hi Jose Maria,
The version used by the taverna-biomoby plugin contains some extra classes.
Once these classes are available in the org.biomoby maven group, the
biomoby.org maven group will be wound down and the plugin will utilize the
official org.biomoby maven group.
Thanks,
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Jos? Mar?a
Fern?ndez Gonz?lez
Sent: July-23-08 6:04 AM
To: Core developer announcements
Subject: Re: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
Hi Eddie and Martin,
I have realized that there are two sets of jMoby artifacts. The
first one
(biomoby.org maven group) is the used by taverna-biomoby plugin, meanwhile
the
second one (org.biomoby maven group) is the officially provided by you.
So, which are the differences between them? The first one is already
in
version 1.2, while the second one has just reached version 1.1.0.
Best Regards,
Jos? Mar?a
Edward Kawas wrote:
> Hi Martin,
>
> Does this update include org.biomoby.service.dashboard.* (the dashboard
> components)?
>
> Thanks,
>
> Eddie
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
> Sent: July-23-08 2:36 AM
> To: Core developer announcements
> Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
>
> I have deployed (to Maven) the current jMoby as version 1.1.0.
>
> Cheers,
> Martin
>
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los
ficheros adjuntos, pueden contener informaci?n protegida para el uso
exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o
cualquier otro tipo de transmisi?n por parte de otra persona que no sea el
destinatario. Si usted recibe por error este correo, se ruega comunicarlo al
remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may
contain confidential and privileged information for the sole use of the
designated recipient named above. Distribution, reproduction or any other
use of this transmission by any party other than the intended recipient is
prohibited. If you are not the intended recipient please contact the sender
and delete all copies.
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From pieter.neerincx at gmail.com Wed Jul 23 11:06:38 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Wed, 23 Jul 2008 17:06:38 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486CA2A7.4070702@bsc.es>
References:
<486CA2A7.4070702@bsc.es>
Message-ID: <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
Hi,
> Martin Senger wrote:
> A service *can* obey such request and send one or more *primitive
> data* as references (the focus on primitive type is new, originaly
> we thought about allowing references on any level, but now, mainly
> becuse of the purpose B we do not propose it anymore).
That doesn't sound very appealing to me. I use quite large data
structures and can have tens of thousands of them in a collection
(think of oligos for micro arrays.) If I can only replace the
primitives with references, I would still have to send a BioMoby XML
structure with hundreds of thousands of references replacing the
primitives. A client or the next service in a workflow would have to
resolve all those references with each reference pointing to only a
relatively small amount of data. That would be a mess and just cause a
huge amount of overhead.
So, I would really love to see the ability to replace a complete
collection or even the entire mobyData block with a reference. Having
said that I really like the idea of purpose "B" too, where I can embed
a link to existing EMBL/Genbank/etc. records on EBI/NCBI/etc. servers.
The only risk would be that a client or the next service in a workflow
happily resolves those links as fast and efficient as it can causing
too much load on those servers. A colleague once managed to have our
entire campus disconnected from the NCBI with simple script resolving
PubMed URLs... it was really easy :). With references it's always
possible to reference something on someone else's servers, but with
purpose "B" it's more likely to happen.
> Martin Senger wrote:
> I do not like using the RDF for mandatory features. The RDF, at
> least in the
> current moby, is optional and should not carry anything that is
> significant
> for the service and its behaviour. The data by reference should be
> part of
> the main BioMoby API (or whatever we call it).
I agree with Martin partially on this one: pass by reference should be
part of the main BioMoby API and not something optional as it is
essential for successful client-service communication. But that
doesn't necessarily exclude using RDF. We could also promote the RDF
stuff from optional to required. I don't want to open a can of worms
here and I'm not advocating making the RDF features mandatory. I'm
just saying it might be an option...
> Dmitry Repchevski wrote:
> If we are going to stay with SOAP (and especially move to doc/lit)
> the right way is to provide streaming through SOAP Attachments (SwA)
> or even better using MTOM.
I don't think attachments of any kind would be a good idea. Although
theoretically a nice solution for streaming data and keeping
everything together, there are practical problems:
* Most firewalls block attachments over 5 MB in size. One of the great
things of web services compared to the good 'ol CORBA is that we can
tunnel over any protocol. So most of us use HTTP(S) to bypass our
institute's paranoid sys admins and skip the part where you'd have to
bug them to open up another firewall port. I'm not sure if this
problem persists with the newer MTOM, but it's disaster with MIME or
DIME attachments.
* I'm not sure about the newer MTOM, but the combination of SOAP and
MIME attachments is a disaster of you're using Perl. The required
modules are dying projects and you'd have to patch them manually to
make it work. If we would go with attachments we would either drop
Perl support for BioMoby or one of us would have to write a SOAP::Lite
+ MIME::tools alternative...
I agree with Dmitry that if we are going to stick with a standard it
has to be a de facto standard. Hence well supported by toolkits/
implementations. A standard that only exists on paper is useless. Most
of us want to build or use BioMoby services and are not that much
interested in writing toolkits. I never managed to become a big WSRF
fan. It's needlessly complicated causing lot's of overhead.
Actually I like what Mark wrote initially most:
> We'll write-up the formal specification of how to do this soon, but
> briefly the idea is that we will use the xlink XML attribute in a
> Moby Object XML <.... xlink=''/>. When that tag is present, it is
> assumed that the content of that node is available at the URI in
> that reference. During service registration a provider will
> indicate the various transport protocols they provide for creating
> references (e.g. http, ftp) and this will be discoverable during a
> registry query. If you don't indicate a protocol, then you are
> saying that you do not support pass-by-reference, and therefore all
> existing services are supported. When accessing a service, you
> indicate to the service provider that you want data to be passed by
> reference by adding an attribute in the mobyData block acceptRefs="http ftp">. The service provider then has the option of
> providing you references for any data they wish. The xlink
> attribute can appear at any level in your Moby object, such that
> some data may be passed in the object itself, while other data from
> the same object may be passed by reference.
For the BioMoby data structures this only requires allowing an
optional xlink attribute and acceptRefs in the request from the
client. Maybe it's nicer to have acceptRefs as a child element of
mobyData or maybe at another level in the XML, but not as an
attribute, because attributes are intended for single items. In the
example above acceptRefs contains a space separated list and off
course we can parse that, but it's ugly XML. Anyway, what Mark wrote
above is very flexible, light-weight and relatively backwards
compatible. If I can choose between a simple <... xlink=""> attribute
or the bulky WSA header from Dmitry's example like in:
http://myserver.com/MyService?asyncId=ID
ID
mobyws:result_queryId00/image/jpeg/content
I'd choose the xlink attribute!
Cheers,
Pi
On 3?Jul?2008, at 11:57 AM, Dmitry Repchevsky wrote:
> Hello,
>
>> For all I know WSRF is already been surpassed by WS-RT. http://www.ibm.com/developerworks/grid/library/gr-wsrfwsrt/index.html?ca=drs-
>> There are quite a number of people who are disenchanted by WSRF.
>> See http://blog.harbulot.com/post/2006/11/22/Experiences-with-WSRF
>> for one example.
>>
>> Machiel
> This is what I am talking about... Martin has a reason we have had
> enough problems with WSRF... now we have a "better" one (WSRT)...
> But (!) There is an implementation? It looks like that this is a
> perfect technology that exists in a paper...
> Should we implement all the standards just to be able to use it? If
> IBM claims to have a new/cool standard (by the way "standard" - who
> is behind it?) but why can't I find (free) implementation of it?
>
> It uses WS-Transfer has more sense to me because it can describe a
> way to represent a moby object in moby message tree (for wsrf). and
> it is at least w3s standard.
> Claiming something to be a standard is always a bold claim...
>
> Dmitry
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
------------------------------------------------------------
From markw at illuminae.com Wed Jul 23 12:55:06 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Wed, 23 Jul 2008 09:55:06 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
Message-ID:
On Wed, 23 Jul 2008 08:06:38 -0700, Pieter Neerincx
wrote:
> That doesn't sound very appealing to me. I use quite large data
> structures and can have tens of thousands of them in a collection (think
> of oligos for micro arrays.) If I can only replace the primitives with
> references, I would still have to send a BioMoby XML structure with
> hundreds of thousands of references replacing the primitives.
Hmmm... that's a situation that we had NEVER considered! Putting xlinks
into a component of the Moby message higher than the mobyData block was
not a scenario that came up during our discussions in Tokyo. the xlink
was really only intended to carry data, not data + moby message
structure... but I suppose it COULD!
How do others feel about this?
M
--
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada
From jmfernandez at cnio.es Wed Jul 23 13:47:54 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Wed, 23 Jul 2008 19:47:54 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
Message-ID: <48876ECA.1040109@cnio.es>
IMHO it can be dangerous for the MOBY payload integrity using XInclude at
mobyContent childs (mobyData) level, but it can be needed in scenarios like
the one described by Pieter. For security reasons, XInclude / XLink / XPointer
should be restricted to the contents of mobyData, not to MOBY Data itself. For
instance, references to Simples or Collections, Collections composed by a
mixure of Simples and references to Simples, or Secondary parameters copied
from run to run.
Best Regards,
Jos? Mar?a
Mark Wilkinson wrote:
> On Wed, 23 Jul 2008 08:06:38 -0700, Pieter Neerincx
> wrote:
>
>
>> That doesn't sound very appealing to me. I use quite large data
>> structures and can have tens of thousands of them in a collection
>> (think of oligos for micro arrays.) If I can only replace the
>> primitives with references, I would still have to send a BioMoby XML
>> structure with hundreds of thousands of references replacing the
>> primitives.
>
> Hmmm... that's a situation that we had NEVER considered! Putting xlinks
> into a component of the Moby message higher than the mobyData block was
> not a scenario that came up during our discussions in Tokyo. the xlink
> was really only intended to carry data, not data + moby message
> structure... but I suppose it COULD!
>
> How do others feel about this?
>
> M
>
>
>
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.
From gordonp at ucalgary.ca Wed Jul 23 14:45:39 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Wed, 23 Jul 2008 12:45:39 -0600
Subject: [MOBY-dev] working with images
In-Reply-To: <4886FB99.1080404@mpiz-koeln.mpg.de>
References: <4886F1E0.5020704@mpiz-koeln.mpg.de> <4886F56C.4090100@toulouse.inra.fr>
<4886FB99.1080404@mpiz-koeln.mpg.de>
Message-ID: <48877C53.7010101@ucalgary.ca>
Hi Andreas,
If you're using the org.biomoby.shared.data package, there is a class
specifically for handling binary data:
http://biomoby.open-bio.org/CVS_CONTENT/moby-live/Java/docs/API/org/biomoby/shared/data/MobyDataBytes.html
Andreas Groscurth wrote:
> Hi,
>
> thanks for that - i forgot to mention i'm looking for a java solution ;-)
> but your hint of encode in Base64 made me google again and found the
> apache.commons.codec and it works.
>
> Thanks for the hint :)
> Andreas
>
> Sebastien Carrere wrote:
>> Hi Andreas,
>>
>> In my web services (written using PlayMoby , then Perl), I use the
>> module MIME::Base64
>> (http://search.cpan.org/~gaas/MIME-Base64-3.07/Base64.pm).
>>
>> I just make a cat on my picture file, and encode the result in
>> Base64, then add it to my Moby String content.
>>
>>
>> Sebastien
>>
>>
>>
>>
>>
>> Andreas Groscurth wrote:
>>> Hi,
>>>
>>> i try to figure out how to transfer images via Web Service in moby....
>>>
>>> E.g. i use the b64_encoded_jpeg. It has the children 'content' which
>>> requires a String, so i need to somehow encode/decode the image file
>>> so that it can be transmitted as a String and the client can read
>>> it as an image (which is somehow done via MobyDataBytes)...
>>>
>>> how to do that ?
>>>
>>> thanks
>>> Andreas
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> !DSPAM:60005,4886f8b9109503161834!
>
>
>
From pieter.neerincx at gmail.com Wed Jul 23 19:48:03 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Thu, 24 Jul 2008 01:48:03 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <48876ECA.1040109@cnio.es>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
Message-ID: <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
Hi Jos?,
I don't understand how references could be security risk. And if they
do why the level at which you could use them would impact security.
Could you please explain?
If references to mobyData blocks are not an option having at least
references to BioMoby articles would already be a big help.
Secondaries are usually very small so I wouldn't need / use references
there, but for primary Simple and especially for primary Collection
articles it would be a big plus!
Cheers,
Pi
On 23 Jul 2008, at 19:47, Jos? Mar?a Fern?ndez Gonz?lez wrote:
> IMHO it can be dangerous for the MOBY payload integrity using
> XInclude at mobyContent childs (mobyData) level, but it can be
> needed in scenarios like the one described by Pieter. For security
> reasons, XInclude / XLink / XPointer should be restricted to the
> contents of mobyData, not to MOBY Data itself. For instance,
> references to Simples or Collections, Collections composed by a
> mixure of Simples and references to Simples, or Secondary parameters
> copied from run to run.
>
> Best Regards,
> Jos? Mar?a
>
> Mark Wilkinson wrote:
>> On Wed, 23 Jul 2008 08:06:38 -0700, Pieter Neerincx > > wrote:
>>> That doesn't sound very appealing to me. I use quite large data
>>> structures and can have tens of thousands of them in a collection
>>> (think of oligos for micro arrays.) If I can only replace the
>>> primitives with references, I would still have to send a BioMoby
>>> XML structure with hundreds of thousands of references replacing
>>> the primitives.
>> Hmmm... that's a situation that we had NEVER considered! Putting
>> xlinks into a component of the Moby message higher than the
>> mobyData block was not a scenario that came up during our
>> discussions in Tokyo. the xlink was really only intended to carry
>> data, not data + moby message structure... but I suppose it COULD!
>> How do others feel about this?
>> M
>
> --
> "There is no reason why anybody would want a computer in their home" -
> Ken Olson, founder of DEC 1977
> "640K ought to be enough for anybody" - Bill Gates, 1981
> "Nobody will ever outgrow a 20Mb hard drive." - ???
>
> "Premature optimization is the root of all evil." - Donald Knuth
>
> Jos? Mar?a Fern?ndez Gonz?lez
> Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
> e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
> Unidad del Instituto Nacional de Bioinform?tica
> Biolog?a Estructural y Biocomputaci?n Structural Biology and
> Biocomputing
> Centro Nacional de Investigaciones Oncol?gicas
> C.P.: 28029 Zip Code: 28029
> C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
>
> **NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso
> los ficheros adjuntos, pueden contener informaci?n protegida para el
> uso exclusivo de su destinatario. Se proh?be la distribuci?n,
> reproducci?n o cualquier otro tipo de transmisi?n por parte de otra
> persona que no sea el destinatario. Si usted recibe por error este
> correo, se ruega comunicarlo al remitente y borrar el mensaje
> recibido.
> **CONFIDENTIALITY NOTICE** This email communication and any
> attachments may contain confidential and privileged information for
> the sole use of the designated recipient named above. Distribution,
> reproduction or any other use of this transmission by any party
> other than the intended recipient is prohibited. If you are not the
> intended recipient please contact the sender and delete all copies.
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
-------------------------------------------------------------
From martin.senger at gmail.com Wed Jul 23 20:41:35 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Thu, 24 Jul 2008 08:41:35 +0800
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <48872C42.3060008@cnio.es>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
<488729f9.20f8720a.0551.0626@mx.google.com> <48872C42.3060008@cnio.es>
Message-ID: <4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com>
> I have realized that there are two sets of jMoby artifacts.
I can't see them. Where should I look?
The official jMoby Maven representation is in
http://biomoby.org/m2repo/org/biomoby/jmoby/ and its headers are:
org.biomoby
jmoby
1.1.0
There is also a Taverna-related artifact, here:
http://biomoby.org/m2repo/org/biomoby/taverna-for-moby/. But it may be too
old and obsolete (version 2004). This one has, indeed, a different group ID
(biomoby.org) which is a bit unfortunate (because it is confusing).
Anything I should change?
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From edward.kawas at gmail.com Wed Jul 23 20:47:45 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 23 Jul 2008 17:47:45 -0700
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com> <488729f9.20f8720a.0551.0626@mx.google.com>
<48872C42.3060008@cnio.es>
<4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com>
Message-ID: <4887d13d.02578c0a.4963.ffffe502@mx.google.com>
Hey Martin,
The one that Jose Maria is referring to is the one on
http://moby.ucalgary.ca/moby_maven/biomoby/org/
That was the one that I was using for the taverna-biomoby plugin.
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
Sent: July-23-08 5:42 PM
To: Core developer announcements
Subject: Re: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
> I have realized that there are two sets of jMoby artifacts.
I can't see them. Where should I look?
The official jMoby Maven representation is in
http://biomoby.org/m2repo/org/biomoby/jmoby/ and its headers are:
org.biomoby
jmoby
1.1.0
There is also a Taverna-related artifact, here:
http://biomoby.org/m2repo/org/biomoby/taverna-for-moby/. But it may be too
old and obsolete (version 2004). This one has, indeed, a different group ID
(biomoby.org) which is a bit unfortunate (because it is confusing).
Anything I should change?
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From martin.senger at gmail.com Wed Jul 23 20:58:02 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Thu, 24 Jul 2008 08:58:02 +0800
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <4887d13d.02578c0a.4963.ffffe502@mx.google.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
<488729f9.20f8720a.0551.0626@mx.google.com> <48872C42.3060008@cnio.es>
<4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com>
<4887d13d.02578c0a.4963.ffffe502@mx.google.com>
Message-ID: <4d93f07c0807231758n6f82f337o2cf7d5ced2d60a76@mail.gmail.com>
>
> The one that Jose Maria is referring to is the one on
> http://moby.ucalgary.ca/moby_maven/biomoby/org/
>
> That was the one that I was using for the taverna-biomoby plugin.
Okay, splendid. Why always must we two, Paul and I, have things
differently/separately? Or perhaps it was not Paul who created this
repository...
We should use one Maven repository that was created for us at open-bio
servers. At least, that is where I am putting jMoby releases. The
taverna-plugin should be there, as well (my opinion).
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From edward.kawas at gmail.com Wed Jul 23 21:01:34 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 23 Jul 2008 18:01:34 -0700
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <4d93f07c0807231758n6f82f337o2cf7d5ced2d60a76@mail.gmail.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com> <488729f9.20f8720a.0551.0626@mx.google.com>
<48872C42.3060008@cnio.es> <4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com> <4887d13d.02578c0a.4963.ffffe502@mx.google.com>
<4d93f07c0807231758n6f82f337o2cf7d5ced2d60a76@mail.gmail.com>
Message-ID: <4887d47a.08b38c0a.380d.fffff846@mx.google.com>
Hey Martin,
Shoot me ;-)
That repository was created by me when taverna moved over to supporting
maven as a way for me to update the core jmoby code that the plugin uses.
Like I said previously, I want to kill that ucalgary repository once the
current one has the code that I need.
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
Sent: July-23-08 5:58 PM
To: Core developer announcements
Subject: Re: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
>
> The one that Jose Maria is referring to is the one on
> http://moby.ucalgary.ca/moby_maven/biomoby/org/
>
> That was the one that I was using for the taverna-biomoby plugin.
Okay, splendid. Why always must we two, Paul and I, have things
differently/separately? Or perhaps it was not Paul who created this
repository...
We should use one Maven repository that was created for us at open-bio
servers. At least, that is where I am putting jMoby releases. The
taverna-plugin should be there, as well (my opinion).
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From markw at illuminae.com Wed Jul 23 21:04:31 2008
From: markw at illuminae.com (markw at illuminae.com)
Date: Thu, 24 Jul 2008 01:04:31 +0000
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <4887d47a.08b38c0a.380d.fffff846@mx.google.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com> <488729f9.20f8720a.0551.0626@mx.google.com><48872C42.3060008@cnio.es> <4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com> <4887d13d.02578c0a.4963.ffffe502@mx.google.com><4d93f07c0807231758n6f82f337o2cf7d5ced2d60a76@mail.gmail.com><4887d47a.08b38c0a.380d.fffff846@mx.google.com>
Message-ID: <2124445639-1216861474-cardhu_decombobulator_blackberry.rim.net-1399312485-@bxe175.bisx.prod.on.blackberry>
Eddie, be careful! Martin might just take you up on the offer! He isn't always as sweet and gentle as he is when he's in the beautiful city of Vancouver! LOL!
We love you Martin :-)
M
On the Road!
-----Original Message-----
From: "Edward Kawas"
Date: Wed, 23 Jul 2008 18:01:34
To: 'Core developer announcements'
Subject: Re: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
Hey Martin,
Shoot me ;-)
That repository was created by me when taverna moved over to supporting
maven as a way for me to update the core jmoby code that the plugin uses.
Like I said previously, I want to kill that ucalgary repository once the
current one has the code that I need.
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
Sent: July-23-08 5:58 PM
To: Core developer announcements
Subject: Re: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
>
> The one that Jose Maria is referring to is the one on
> http://moby.ucalgary.ca/moby_maven/biomoby/org/
>
> That was the one that I was using for the taverna-biomoby plugin.
Okay, splendid. Why always must we two, Paul and I, have things
differently/separately? Or perhaps it was not Paul who created this
repository...
We should use one Maven repository that was created for us at open-bio
servers. At least, that is where I am putting jMoby releases. The
taverna-plugin should be there, as well (my opinion).
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From martin.senger at gmail.com Wed Jul 23 21:35:59 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Thu, 24 Jul 2008 09:35:59 +0800
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <2124445639-1216861474-cardhu_decombobulator_blackberry.rim.net-1399312485-@bxe175.bisx.prod.on.blackberry>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
<488729f9.20f8720a.0551.0626@mx.google.com> <48872C42.3060008@cnio.es>
<4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com>
<4887d13d.02578c0a.4963.ffffe502@mx.google.com>
<4d93f07c0807231758n6f82f337o2cf7d5ced2d60a76@mail.gmail.com>
<4887d47a.08b38c0a.380d.fffff846@mx.google.com>
<2124445639-1216861474-cardhu_decombobulator_blackberry.rim.net-1399312485-@bxe175.bisx.prod.on.blackberry>
Message-ID: <4d93f07c0807231835y3b9457d2q65728de32d7b6c17@mail.gmail.com>
:-) - I promise: no shooting!
Thanks for all clarification (and, BTW, thanks for wonderful days in
Vancouver, I really, really, enjoyed it).
Cheers,
Martin
PS. Eddie, let me know if you need any assistance with the [re]packaging of
the jMoby libraries. M.
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From jmfernandez at cnio.es Thu Jul 24 09:34:05 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Thu, 24 Jul 2008 15:34:05 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
Message-ID: <488884CD.9030403@cnio.es>
Hi Pieter,
I agree with you about the references to primary Simple and Collection
articles: they are absolutely needed. References to Secondary parameters can
be needed to copy/use the same values in other calls. I am not strongly
against references to full mobyData elements, but they should be considered
and used very carefully, because they can add more mobyData elements than you
wanted, and you cannot check duplicity of queryIds until all the referenced
mobyData elements are fetched.
With an ill-designed service (or libraries) and a carefully built XInclude
reference, with no checks local contents (from the computer or the intranet)
could be referenced and perhaps obtained. With an even more ill-designed
service (or libraries) and a carefully built XInclude reference, an external
hacker could fire OOM (Out Of Memory) or buffer overflow conditions.
Best Regards,
Jos? Mar?a
Pieter Neerincx wrote:
> Hi Jos?,
>
> I don't understand how references could be security risk. And if they do
> why the level at which you could use them would impact security. Could
> you please explain?
>
> If references to mobyData blocks are not an option having at least
> references to BioMoby articles would already be a big help. Secondaries
> are usually very small so I wouldn't need / use references there, but
> for primary Simple and especially for primary Collection articles it
> would be a big plus!
>
> Cheers,
>
> Pi
>
> On 23 Jul 2008, at 19:47, Jos? Mar?a Fern?ndez Gonz?lez wrote:
>
>> IMHO it can be dangerous for the MOBY payload integrity using XInclude
>> at mobyContent childs (mobyData) level, but it can be needed in
>> scenarios like the one described by Pieter. For security reasons,
>> XInclude / XLink / XPointer should be restricted to the contents of
>> mobyData, not to MOBY Data itself. For instance, references to Simples
>> or Collections, Collections composed by a mixure of Simples and
>> references to Simples, or Secondary parameters copied from run to run.
>>
>> Best Regards,
>> Jos? Mar?a
>>
>> Mark Wilkinson wrote:
>>> On Wed, 23 Jul 2008 08:06:38 -0700, Pieter Neerincx
>>> wrote:
>>>> That doesn't sound very appealing to me. I use quite large data
>>>> structures and can have tens of thousands of them in a collection
>>>> (think of oligos for micro arrays.) If I can only replace the
>>>> primitives with references, I would still have to send a BioMoby XML
>>>> structure with hundreds of thousands of references replacing the
>>>> primitives.
>>> Hmmm... that's a situation that we had NEVER considered! Putting
>>> xlinks into a component of the Moby message higher than the mobyData
>>> block was not a scenario that came up during our discussions in
>>> Tokyo. the xlink was really only intended to carry data, not data +
>>> moby message structure... but I suppose it COULD!
>>> How do others feel about this?
>>> M
>>
>> --
>> "There is no reason why anybody would want a computer in their home" -
>> Ken Olson, founder of DEC 1977
>> "640K ought to be enough for anybody" - Bill Gates, 1981
>> "Nobody will ever outgrow a 20Mb hard drive." - ???
>>
>> "Premature optimization is the root of all evil." - Donald Knuth
>>
>> Jos? Mar?a Fern?ndez Gonz?lez
>> Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
>> e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
>> Unidad del Instituto Nacional de Bioinform?tica
>> Biolog?a Estructural y Biocomputaci?n Structural Biology and
>> Biocomputing
>> Centro Nacional de Investigaciones Oncol?gicas
>> C.P.: 28029 Zip Code: 28029
>> C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
>>
>> **NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los
>> ficheros adjuntos, pueden contener informaci?n protegida para el uso
>> exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n
>> o cualquier otro tipo de transmisi?n por parte de otra persona que no
>> sea el destinatario. Si usted recibe por error este correo, se ruega
>> comunicarlo al remitente y borrar el mensaje recibido.
>> **CONFIDENTIALITY NOTICE** This email communication and any
>> attachments may contain confidential and privileged information for
>> the sole use of the designated recipient named above. Distribution,
>> reproduction or any other use of this transmission by any party other
>> than the intended recipient is prohibited. If you are not the intended
>> recipient please contact the sender and delete all copies.
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> -------------------------------------------------------------
> Wageningen University and Research centre (WUR)
> Laboratory of Bioinformatics
> Transitorium (building 312) room 1034
>
> Dreijenlaan 3
> 6703 HA Wageningen
> The Netherlands
>
> phone: +31 (0)317-483 060
> mobile: +31 (0)6-143 66 783
> e-mail: pieter.neerincx at gmail.com
> skype: pieter.online
> -------------------------------------------------------------
>
>
>
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.
From martin.senger at gmail.com Thu Jul 24 20:06:03 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Fri, 25 Jul 2008 08:06:03 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <488884CD.9030403@cnio.es>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
Message-ID: <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
Thank you for all your comments. We need to move on, however. I am, almost
impatiently, waiting for a conclusion because I need to implement the
"data-by-reference" rather soon. Are there other people preparing/building
their comments - so we should wait for them, or can I summarize what was
said so far - and create an official "RFC" from it?
My personal opinion is still on the side to have references only for
primitives. I am not convinced by the Peter's vision of a huge collection
that is too big even when it uses references. I am simply trying to find a
solution that will be good for the most cases - and then deal with the
marginal cases separately (like by designing the service differently).
I would like to hear also Eddie's voice - because he knows how easy or hard
it would be to make the way we decide to do the "data-by-reference" in
Taverna (and I am still talking about the T1 which I expect to be supported
for some time).
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Fri Jul 25 05:25:31 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Fri, 25 Jul 2008 11:25:31 +0200
Subject: [MOBY-dev] findService and the hierarchy
In-Reply-To: <4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
References: <487C9061.3030203@mpiz-koeln.mpg.de>
<4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
Message-ID: <48899C0B.70408@mpiz-koeln.mpg.de>
i like to bring this up for discussion again ;-)
thanks
Martin Senger wrote:
>> In my opinion when searching for services consuming Annotated_GO_Term via
>> the findService method i should also include the service which consumes a
>> GO_Term datatype, as this is the parent datatype (per default
>> 'includeParentDatatype' is true)....
>>
>>
>
> I assume that by "I should also include" you actually meant "the registry
> response should also include". Is my understanding correct?
>
> If so, I am afraid that it is not the issue of the jMoby library but the
> registry itself. Eddie or Mark can confirm or reject this hypothesis.
>
> The BioMoby API allows to specify two XML tags when calling findService
> method: the
>
> X
> and
> X
>
> Where X can be either 1 or 0.
>
> The jMoby's findService() method sets these two tags according to the two
> boolean parameters that the method can get:
>
> boolean includeChildrenServiceTypes
> and
> boolean includeParentDataTypes
>
> I have tried to call the findService, with the Annotated_GO_Term input,
> using all combinations of these two tags (you can do it using the Moby
> command-line client with the optioins -nc and/or -np). The results are
> either no service or 713 services:
>
> expandObjects expandServices # of found services
> 1 1 713
> 0 0 0
> 1 0 713
> 0 0 0
>
> I have feeling that this is not a correct result. Could Eddie or Mark
> perhaps comment on it?
>
> [I vaguely remember that we had this conversation about these tags with Mark
> years(!) ago - but I do not recall what was the conclusion. I only, and
> again vaguely, remember to have this issue in my to-do list - which would
> indicate that something could be done in the jMoby library.]
>
> Cheers,
> Martin
>
>
--
=====================================================
Dipl. Bioinf. Andreas Groscurth
Bioinformatics Software Developer
Plant Computational Biology group
Max-Planck Institute for plant breeding research
Carl-von-Linne Weg 10
50829 Cologne
Germany
+49(0) 221 5062449
=====================================================
From martin.senger at gmail.com Fri Jul 25 05:34:15 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Fri, 25 Jul 2008 17:34:15 +0800
Subject: [MOBY-dev] findService and the hierarchy
In-Reply-To: <48899C0B.70408@mpiz-koeln.mpg.de>
References: <487C9061.3030203@mpiz-koeln.mpg.de>
<4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
<48899C0B.70408@mpiz-koeln.mpg.de>
Message-ID: <4d93f07c0807250234h53e049bv9b97edecbd0865e3@mail.gmail.com>
> i like to bring this up for discussion again ;-)
Well, let's not discuss it but let's look what we are getting back from the
registry when different values are given for expandObjects and
expandService. I posted my results - now it is up to Eddie or Mark to draw
conclusion from that results... :-)
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From pieter.neerincx at gmail.com Fri Jul 25 06:50:12 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Fri, 25 Jul 2008 12:50:12 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
Message-ID: <2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
Hi Martin at al.,
On 25?Jul?2008, at 2:06 AM, Martin Senger wrote:
> Thank you for all your comments. We need to move on, however. I am,
> almost
> impatiently, waiting for a conclusion because I need to implement the
> "data-by-reference" rather soon. Are there other people preparing/
> building
> their comments - so we should wait for them, or can I summarize what
> was
> said so far - and create an official "RFC" from it?
>
> My personal opinion is still on the side to have references only for
> primitives. I am not convinced by the Pieter's vision of a huge
> collection
> that is too big even when it uses references. I am simply trying to
> find a
> solution that will be good for the most cases - and then deal with the
> marginal cases separately (like by designing the service differently).
Well, especially if you want to have a solution that works in most
cases, I think we should have pass-by-reference at *any* level in
mobyData. How often will the data of a single primitive be too large
for inclusion in the SOAP body? I can imagine wrapping legacy data
formats like a complete EMBL/Genbank/DDBJ record in a String object or
sending images as base 64 encoded pieces of text in a String object.
Maybe even sending a bulky sequence like a complete chromosome inside
a String object, but in most cases the raw data carried by a primitive
will be smaller than your average URL! And even if you do send some
bulky stuff inside a String object as long as you send only one of
those at a time it'll still work in most cases. But once you start to
send tens, hundreds or more of those you have a problem. If you still
send around tens, hundreds or more URLs, you'd have to fetch each one
individually causing massive overhead. The reason to implement pass-by-
reference is to make BioMoby scale much better for big jobs and in
such cases fetching data from a single reference is much more efficient.
Take for example position information of alignments on a genome:
13
97419218
97419282
65M
With pass-by-reference only for primitives this would become something
like this:
That ain't much of an improvement!
I have on average 30.000 features on a micro array and for each oligo
on average 3 hits on a reference assembly. That makes for an average
total of 90.000 HitPosition objects and that is just a small part of
the annotation for my oligos....
Just to stress that these are not hypothetical cases: The stuff above
is just a single example of what I'm already using for more than two
years. Off course I needed a pass by reference workaround, because the
data is too big for the SOAP body. So I registered an URL object and
send those around. These URL objects point to chunks of BioMoby XML
which is a complete mobyData block in most cases. Although the URL
obeject is fully compatible with the current BioMoby standard, it's an
ugly solution because of two reasons:
1. It's not a standard to do pass-by-reference.
2. It defeats the entire purpose of having the BioMoby object ontology
to improve automatic service discovery. You can only discover that I
provide several services which consume or produce URL objects, but you
can not discover automatically what those URLs point to, so most of
the URL object producers and consumers will be incompatible! (Ok, I
can use namespace restriction to limit the problem of incompatible
services a bit, but still you would have no idea what the URL points
to based on the data in BioMoby Central.)
So I would love to see standardised pass-by-reference as part of the
BioMoby specs and I think it doesn't require rocket science to do this
at any level in the structure of BioMoby objects. Why don't we simply
do the following:
Current situation:
A BioMoby object is a BioMoby tripple with optional articleName
attribute and optionally raw character data for primitives. The
tripple is the XML element name, an id attribute and a namespace
attribute.
New situation:
The above + The id and namespace attributes of a BioMoby tripple can
be replaced with an xlink attribute resulting in a "BioMoby double".
If the latter is the case the element containing the xlink attribute
and all it's children are available from link specified by the xlink
attribute.
Example old:
ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
4569343
Example with pass-by-reference:
which points to:
ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
4569343
Hence note that the link points to a ComplexObject and not just to
it's children. The latter would also be an option but than you would
get:
which points to:
ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
4569343
If you have more than one child element like with the example above,
some XML parsers might have problems with such a chunk. Although it is
well balanced, it doesn't have a (pseudo-)root element. So for
practical reasons I suggest to have the links point to the element
containing the xlink attribute and it's children. This should be
really easy to parse. You either have an id + namespace attribute or
you have an xlink attribute. If both were not present the values were
NULL. This doesn't require parsing dozens of WSRF tags in a header. It
even doesn't require a service to tell the client using the
serviceNotes or something similar that it did pass-by-reference of
some kind nor does it require the client to specify that it can
understand certain references. Off course it would be handy though to
have some extension to BioMoby Central to prevent discovering services
providing references which are incompatible with your client.
Secondly, if a service can provide multiple types of references and
your client doesn't understand them all, it would also be nice if a
client can specify a preference for a certain type of reference. But
both wouldn't be required for a first quick implementation of pass-by-
reference.
> I would like to hear also Eddie's voice - because he knows how easy
> or hard
> it would be to make the way we decide to do the "data-by-reference" in
> Taverna (and I am still talking about the T1 which I expect to be
> supported
> for some time).
I agree we need Eddie's feedback on Taverna compatibility!
Compatibility with Taverna 1 would be great, but it would be extremely
lame if our BioMoby references turn out to be incompatible with the
new pass-by-reference feature of Taverna 2.
I hope you see the potential for improving scalability of BioMoby
services with pass-by-reference at any level of mobyData!
Cheers,
Pi
> Martin
>
> --
> Martin Senger
> email: martin.senger at gmail.com,m.senger at cgiar.org
> skype: martinsenger
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
------------------------------------------------------------
From martin.senger at gmail.com Fri Jul 25 07:14:01 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Fri, 25 Jul 2008 19:14:01 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
Message-ID: <4d93f07c0807250414q1d456695ld65cd3d5a6daa63@mail.gmail.com>
Pieter, thanks! Your reply was very good, and convincing! I like the most
the fact that you are driven by a real use case, and a long one, lasting
already for two years.
I had in mind other use cases, but now I agree that Pieter's point of view
is better. I will read his email once again, just to be sure that I have not
missed something.
Are we closer now to the RFC proposal?
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Fri Jul 25 07:15:52 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Fri, 25 Jul 2008 13:15:52 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es> <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com> <488884CD.9030403@cnio.es> <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
Message-ID: <4889B5E8.6010903@mpiz-koeln.mpg.de>
Hi,
I'm not sure whether i get the complete point of the discussion here,
nevertheless I like to jump in into that.
What i dont understand is why do we need references for only primitives
? I dont understand what a reference for a primitive is good for ?
I would also like to have references for any type, if possible.
Currently i work on sending image information around. some images are up
to several MBs. It would be way more convinient to send references to
the http url of the image instead of encoding/decoding it and
transferring around some networks.
If any type of object could be a reference, then we could stay with the
current way of registering and discovering services....
It might also just the case i dont get the point what martin said ;-)
Cheers
andreas
Pieter Neerincx wrote:
> Hi Martin at al.,
>
> On 25?Jul?2008, at 2:06 AM, Martin Senger wrote:
>
>> Thank you for all your comments. We need to move on, however. I am,
>> almost
>> impatiently, waiting for a conclusion because I need to implement the
>> "data-by-reference" rather soon. Are there other people
>> preparing/building
>> their comments - so we should wait for them, or can I summarize what was
>> said so far - and create an official "RFC" from it?
>>
>> My personal opinion is still on the side to have references only for
>> primitives. I am not convinced by the Pieter's vision of a huge
>> collection
>> that is too big even when it uses references. I am simply trying to
>> find a
>> solution that will be good for the most cases - and then deal with the
>> marginal cases separately (like by designing the service differently).
>
> Well, especially if you want to have a solution that works in most
> cases, I think we should have pass-by-reference at *any* level in
> mobyData. How often will the data of a single primitive be too large
> for inclusion in the SOAP body? I can imagine wrapping legacy data
> formats like a complete EMBL/Genbank/DDBJ record in a String object or
> sending images as base 64 encoded pieces of text in a String object.
> Maybe even sending a bulky sequence like a complete chromosome inside
> a String object, but in most cases the raw data carried by a primitive
> will be smaller than your average URL! And even if you do send some
> bulky stuff inside a String object as long as you send only one of
> those at a time it'll still work in most cases. But once you start to
> send tens, hundreds or more of those you have a problem. If you still
> send around tens, hundreds or more URLs, you'd have to fetch each one
> individually causing massive overhead. The reason to implement
> pass-by-reference is to make BioMoby scale much better for big jobs
> and in such cases fetching data from a single reference is much more
> efficient.
>
> Take for example position information of alignments on a genome:
>
> moby:namespace='transcriptome' moby:articleName='hit_position'>
> moby:articleName='seq_id'> 13
> moby:articleName='start'> 97419218
> moby:articleName='stop'> 97419282
> moby:articleName='strand'/>
> moby:articleName='cll'> 65M
>
>
> With pass-by-reference only for primitives this would become something
> like this:
>
> moby:namespace='transcriptome' moby:articleName='hit_position'>
> moby:articleName='seq_id'
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/String_seq_id.xml'/>
>
> moby:articleName='start'
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/Integer_start.xml'/>
>
> moby:articleName='stop'
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/Integer_stop.xml'/>
> moby:articleName='strand'/>
> moby:articleName='cll'
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/String_cll.xml'/>
>
>
> That ain't much of an improvement!
>
> I have on average 30.000 features on a micro array and for each oligo
> on average 3 hits on a reference assembly. That makes for an average
> total of 90.000 HitPosition objects and that is just a small part of
> the annotation for my oligos....
>
> Just to stress that these are not hypothetical cases: The stuff above
> is just a single example of what I'm already using for more than two
> years. Off course I needed a pass by reference workaround, because the
> data is too big for the SOAP body. So I registered an URL object and
> send those around. These URL objects point to chunks of BioMoby XML
> which is a complete mobyData block in most cases. Although the URL
> obeject is fully compatible with the current BioMoby standard, it's an
> ugly solution because of two reasons:
> 1. It's not a standard to do pass-by-reference.
> 2. It defeats the entire purpose of having the BioMoby object ontology
> to improve automatic service discovery. You can only discover that I
> provide several services which consume or produce URL objects, but you
> can not discover automatically what those URLs point to, so most of
> the URL object producers and consumers will be incompatible! (Ok, I
> can use namespace restriction to limit the problem of incompatible
> services a bit, but still you would have no idea what the URL points
> to based on the data in BioMoby Central.)
>
> So I would love to see standardised pass-by-reference as part of the
> BioMoby specs and I think it doesn't require rocket science to do this
> at any level in the structure of BioMoby objects. Why don't we simply
> do the following:
>
> Current situation:
>
> A BioMoby object is a BioMoby tripple with optional articleName
> attribute and optionally raw character data for primitives. The
> tripple is the XML element name, an id attribute and a namespace
> attribute.
>
> New situation:
> The above + The id and namespace attributes of a BioMoby tripple can
> be replaced with an xlink attribute resulting in a "BioMoby double".
> If the latter is the case the element containing the xlink attribute
> and all it's children are available from link specified by the xlink
> attribute.
>
> Example old:
>
> articleName='MyFavoriteObject'>
>
> ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
>
>
> 4569343
>
>
>
> Example with pass-by-reference:
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/ComplexObject.xml' articleName='MyFavoriteObject'
> />
>
> which points to:
>
> articleName='MyFavoriteObject'>
>
> ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
>
>
> 4569343
>
>
>
> Hence note that the link points to a ComplexObject and not just to
> it's children. The latter would also be an option but than you would get:
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/ComplexObject_content.xml' id='accession_number123875'
> namespace='' articleName='MyFavoriteObject' />
>
> which points to:
>
>
> ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
>
>
> 4569343
>
>
> If you have more than one child element like with the example above,
> some XML parsers might have problems with such a chunk. Although it is
> well balanced, it doesn't have a (pseudo-)root element. So for
> practical reasons I suggest to have the links point to the element
> containing the xlink attribute and it's children. This should be
> really easy to parse. You either have an id + namespace attribute or
> you have an xlink attribute. If both were not present the values were
> NULL. This doesn't require parsing dozens of WSRF tags in a header. It
> even doesn't require a service to tell the client using the
> serviceNotes or something similar that it did pass-by-reference of
> some kind nor does it require the client to specify that it can
> understand certain references. Off course it would be handy though to
> have some extension to BioMoby Central to prevent discovering services
> providing references which are incompatible with your client.
> Secondly, if a service can provide multiple types of references and
> your client doesn't understand them all, it would also be nice if a
> client can specify a preference for a certain type of reference. But
> both wouldn't be required for a first quick implementation of
> pass-by-reference.
>
>> I would like to hear also Eddie's voice - because he knows how easy
>> or hard
>> it would be to make the way we decide to do the "data-by-reference" in
>> Taverna (and I am still talking about the T1 which I expect to be
>> supported
>> for some time).
>
> I agree we need Eddie's feedback on Taverna compatibility!
> Compatibility with Taverna 1 would be great, but it would be extremely
> lame if our BioMoby references turn out to be incompatible with the
> new pass-by-reference feature of Taverna 2.
>
> I hope you see the potential for improving scalability of BioMoby
> services with pass-by-reference at any level of mobyData!
>
> Cheers,
>
> Pi
>
>> Martin
>>
>> --
>> Martin Senger
>> email: martin.senger at gmail.com,m.senger at cgiar.org
>> skype: martinsenger
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> -------------------------------------------------------------
> Wageningen University and Research centre (WUR)
> Laboratory of Bioinformatics
> Transitorium (building 312) room 1034
>
> Dreijenlaan 3
> 6703 HA Wageningen
> The Netherlands
>
> phone: +31 (0)317-483 060
> mobile: +31 (0)6-143 66 783
> e-mail: pieter.neerincx at gmail.com
> skype: pieter.online
> ------------------------------------------------------------
>
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
--
=====================================================
Dipl. Bioinf. Andreas Groscurth
Bioinformatics Software Developer
Plant Computational Biology group
Max-Planck Institute for plant breeding research
Carl-von-Linne Weg 10
50829 Cologne
Germany
+49(0) 221 5062449
=====================================================
From martin.senger at gmail.com Fri Jul 25 07:35:33 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Fri, 25 Jul 2008 19:35:33 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4889B5E8.6010903@mpiz-koeln.mpg.de>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
<4889B5E8.6010903@mpiz-koeln.mpg.de>
Message-ID: <4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
> What i dont understand is why do we need references for only primitives ? I
> dont understand what a reference for a primitive is good for ?
If you do not understand, it is my fault - I have tried to explain it, but
obviously have not done it well.
The discussion is now, however, a bit farer.Your first question is probably
obsolete - the forming RFC seems to be that the references will be able to
be anywhere (or not only on primitives).
Your second question: We need to allow reference on primitives if we want to
reuse existing references on not-yet-moby objects (such as uniprot
sequences).
The same with URLs of existing images - as in your use case. What you are
describing is exactly what we are going to do. You will have a choice: a)
You store somewhere an XML moby message (its part) and send a reference to
it, or b) you store somewhere only the image (not wrapped in a moby message)
and you send a moby message with a reference attached to a primitive.
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Fri Jul 25 07:46:45 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Fri, 25 Jul 2008 13:46:45 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es> <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com> <488884CD.9030403@cnio.es> <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com> <2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com> <4889B5E8.6010903@mpiz-koeln.mpg.de>
<4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
Message-ID: <4889BD25.3080701@mpiz-koeln.mpg.de>
Martin Senger wrote:
> If you do not understand, it is my fault - I have tried to explain it, but
> obviously have not done it well.
>
hehe not automatically - i might just missed something ;-)
Thanks for the explanation, now i know i did understand it correctly ;-)
Andreas
> The discussion is now, however, a bit farer.Your first question is probably
> obsolete - the forming RFC seems to be that the references will be able to
> be anywhere (or not only on primitives).
>
> Your second question: We need to allow reference on primitives if we want to
> reuse existing references on not-yet-moby objects (such as uniprot
> sequences).
>
> The same with URLs of existing images - as in your use case. What you are
> describing is exactly what we are going to do. You will have a choice: a)
> You store somewhere an XML moby message (its part) and send a reference to
> it, or b) you store somewhere only the image (not wrapped in a moby message)
> and you send a moby message with a reference attached to a primitive.
>
> Martin
>
>
--
=====================================================
Dipl. Bioinf. Andreas Groscurth
Bioinformatics Software Developer
Plant Computational Biology group
Max-Planck Institute for plant breeding research
Carl-von-Linne Weg 10
50829 Cologne
Germany
+49(0) 221 5062449
=====================================================
From pieter.neerincx at gmail.com Fri Jul 25 08:28:43 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Fri, 25 Jul 2008 14:28:43 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
<4889B5E8.6010903@mpiz-koeln.mpg.de>
<4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
Message-ID:
Hi,
On 25?Jul?2008, at 1:35 PM, Martin Senger wrote:
>> What i dont understand is why do we need references for only
>> primitives ? I
>> dont understand what a reference for a primitive is good for ?
>
>
> If you do not understand, it is my fault - I have tried to explain
> it, but
> obviously have not done it well.
>
> The discussion is now, however, a bit farer.Your first question is
> probably
> obsolete - the forming RFC seems to be that the references will be
> able to
> be anywhere (or not only on primitives).
>
> Your second question: We need to allow reference on primitives if we
> want to
> reuse existing references on not-yet-moby objects (such as uniprot
> sequences).
Yes :) !
> The same with URLs of existing images - as in your use case. What
> you are
> describing is exactly what we are going to do. You will have a
> choice: a)
> You store somewhere an XML moby message (its part) and send a
> reference to
> it, or b) you store somewhere only the image (not wrapped in a moby
> message)
> and you send a moby message with a reference attached to a primitive.
Eh, but in the case of images they must be encoded to put then inside
a String primitive. So, since the data the reference points to must be
the same as in the case where you would not use a reference, it musty
also be encoded. Hence, you can not replace the object with a
reference to a plain (un-encoded) JPEG for example. Take for example
the b64_encoded_jpeg Object. That would look like:
afr78e349tu9rghjerewptu59hgfirhegwug9uwguwujw9-
If you want to replace this with a reference the actual data would
still have to be a base64 encoded piece of text. So you can not make a
reference to an image that exists as a plain JPEG, GIF, etc. on a web
server. We don't have a datatype to send un-encoded binary stuff. You
could create a new (un-encoded) JPEG object and use references, but
than you are required to always send a reference as you can not send
that BioMoby datatype un-encoded as part of the SOAP payload! This is
again not optimal though as you would have different objects for
encoded and un-encoded images limiting the discovery of potentially
compatible services. I couldn't care less about whether it's encoded
or not. I just want to send or receive an image and the optional
encoding is just the side effect of a workaround. I don't see how we
can easily solve that. Does anyone have suggestion how we should
handle this?...
Cheers,
Pi
> Martin
>
> --
> Martin Senger
> email: martin.senger at gmail.com,m.senger at cgiar.org
> skype: martinsenger
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
------------------------------------------------------------
From edward.kawas at gmail.com Fri Jul 25 09:03:34 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Fri, 25 Jul 2008 06:03:34 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es> <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com> <488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
Message-ID: <4889cf38.1aba720a.1b32.3a36@mx.google.com>
Hi Martin,
Don't worry about Taverna! Whatever we decide will most likely be the same
amount of work. So let's just get it right.
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
Sent: July-24-08 5:06 PM
To: Core developer announcements
Subject: Re: [MOBY-dev] data by reference - a request for comments
Thank you for all your comments. We need to move on, however. I am, almost
impatiently, waiting for a conclusion because I need to implement the
"data-by-reference" rather soon. Are there other people preparing/building
their comments - so we should wait for them, or can I summarize what was
said so far - and create an official "RFC" from it?
My personal opinion is still on the side to have references only for
primitives. I am not convinced by the Peter's vision of a huge collection
that is too big even when it uses references. I am simply trying to find a
solution that will be good for the most cases - and then deal with the
marginal cases separately (like by designing the service differently).
I would like to hear also Eddie's voice - because he knows how easy or hard
it would be to make the way we decide to do the "data-by-reference" in
Taverna (and I am still talking about the T1 which I expect to be supported
for some time).
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From gordonp at ucalgary.ca Fri Jul 25 11:05:42 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Fri, 25 Jul 2008 09:05:42 -0600
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4889cf38.1aba720a.1b32.3a36@mx.google.com>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es> <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com> <488884CD.9030403@cnio.es> <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<4889cf38.1aba720a.1b32.3a36@mx.google.com>
Message-ID: <4889EBC6.6010500@ucalgary.ca>
I'll finally chime in...I'm in favour of data-by-reference at any level
within a mobyData block. Above that and you get weird issues with
exception maintenance, and any restriction below that appears pretty
artificial to me.
Edward Kawas wrote:
> Hi Martin,
>
> Don't worry about Taverna! Whatever we decide will most likely be the same
> amount of work. So let's just get it right.
>
> Eddie
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
> Sent: July-24-08 5:06 PM
> To: Core developer announcements
> Subject: Re: [MOBY-dev] data by reference - a request for comments
>
> Thank you for all your comments. We need to move on, however. I am, almost
> impatiently, waiting for a conclusion because I need to implement the
> "data-by-reference" rather soon. Are there other people preparing/building
> their comments - so we should wait for them, or can I summarize what was
> said so far - and create an official "RFC" from it?
>
> My personal opinion is still on the side to have references only for
> primitives. I am not convinced by the Peter's vision of a huge collection
> that is too big even when it uses references. I am simply trying to find a
> solution that will be good for the most cases - and then deal with the
> marginal cases separately (like by designing the service differently).
>
> I would like to hear also Eddie's voice - because he knows how easy or hard
> it would be to make the way we decide to do the "data-by-reference" in
> Taverna (and I am still talking about the T1 which I expect to be supported
> for some time).
>
> Martin
>
>
From jason.e.stewart at gmail.com Fri Jul 25 12:07:22 2008
From: jason.e.stewart at gmail.com (Jason Stewart)
Date: Fri, 25 Jul 2008 21:37:22 +0530
Subject: [MOBY-dev] Interesting...
In-Reply-To:
References:
Message-ID: <41c1ade50807250907k61a3a459m3d394bf6d793df5f@mail.gmail.com>
Hey Mark,
On Wed, Jul 2, 2008 at 7:52 PM, Mark wrote:
> http://www.biomedcentral.com/1471-2105/9/295/abstract
>
> I wonder how they got document-literal stuff working in Perl??
Can you get me a pdf of that article? It looks like a subscription is needed.
Cheers, jas.
From edward.kawas at gmail.com Fri Jul 25 14:05:23 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Fri, 25 Jul 2008 11:05:23 -0700
Subject: [MOBY-dev] updated wsrf perl module
Message-ID: <488a1605.06d7720a.021c.ffff9cd4@mx.google.com>
Hi all,
I am trying to create a cpan distribution for the WSRF module created by
Mark McKeown. In the process, I wanted to see if I could get it to work with
windows (you never know, there might be some windows moby service developers
out there somewhere ;-) ) and to see if I could improve the install
procedure.
So, I now have a beta version of the module that I plan on sending to cpan,
but I want to make sure that I didn't break anything for those using this
module for creating async services. Is it possible for those that have
developed async services to try out this new module and to let me know if
anything is missing, broken, etc?
I would really appreciate any feedback.
Thanks,
Eddie
-------------- next part --------------
A non-text attachment was scrubbed...
Name: WSRF-Lite-0.8.2.2.1.tar.gz
Type: application/octet-stream
Size: 454887 bytes
Desc: not available
URL:
From markw at illuminae.com Fri Jul 25 14:34:13 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Fri, 25 Jul 2008 11:34:13 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4889EBC6.6010500@ucalgary.ca>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<4889cf38.1aba720a.1b32.3a36@mx.google.com>
<4889EBC6.6010500@ucalgary.ca>
Message-ID:
On Fri, 25 Jul 2008 08:05:42 -0700, Paul Gordon
wrote:
> I'll finally chime in...I'm in favour of data-by-reference at any level
> within a mobyData block. Above that and you get weird issues with
> exception maintenance, and any restriction below that appears pretty
> artificial to me.
I agree with Paul. I wouldn't go any higher than the mobyData block, but
allowing it at any lower levels seems sensible to me, particularly given
Pieter's use-case.
M
--
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada
From markw at illuminae.com Fri Jul 25 14:33:07 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Fri, 25 Jul 2008 11:33:07 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
<4889B5E8.6010903@mpiz-koeln.mpg.de>
<4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
Message-ID:
On Fri, 25 Jul 2008 05:28:43 -0700, Pieter Neerincx
wrote:
> If you want to replace this with a reference the actual data would still
> have to be a base64 encoded piece of text. So you can not make a
> reference to an image that exists as a plain JPEG, GIF, etc. on a web
> server.
Martin and I discussed this while he was here and the behaviour of this
case will be part of his RFC proposal. It seems to me that it would be
silly to b64 encode something that is being passed by reference... but I
agree that to do anything else is a 'hack'. Nevertheless, there are a lot
of 'hacks' in Moby, and this one is one of the more sensible 'hacks' IMO...
M
--
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada
From jason.e.stewart at gmail.com Sat Jul 26 01:52:11 2008
From: jason.e.stewart at gmail.com (Jason Stewart)
Date: Sat, 26 Jul 2008 11:22:11 +0530
Subject: [MOBY-dev] updated wsrf perl module
In-Reply-To: <488a1605.06d7720a.021c.ffff9cd4@mx.google.com>
References: <488a1605.06d7720a.021c.ffff9cd4@mx.google.com>
Message-ID: <41c1ade50807252252r2cc2482euad274ab511dd3421@mail.gmail.com>
Hey,
Cool Eddie - it's a pain to have to install perl modules by hand -
glad you're making this available on CPAN. I don't have any async
services though...
Cheers, jas.
On Fri, Jul 25, 2008 at 11:35 PM, Edward Kawas wrote:
> Hi all,
>
> I am trying to create a cpan distribution for the WSRF module created by
> Mark McKeown.
From dmitry.repchevski at bsc.es Sat Jul 26 06:49:07 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Sat, 26 Jul 2008 12:49:07 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
Message-ID: <488B0123.3040608@bsc.es>
Hello everybody,
WSRF has a way to reference a part of resource using XPath expressions.
It's easy to include it just by updating current asynchronous specification.
The only thing to do is to implement QueryResourceProperties method.
So we can always get any part of mobyData (for practical reason I think
to limit it to MobySimple/MobyCollection).
This way the reference to any remote resource will be standard WSRF
reference (based on WS-Addressing) with XPath expression to get the
interested part of it.
I'm not very good in XPath, but something like this:
String expression =
"/*[local-name()='MOBY']/*[local-name()='mobyContent']/*[local-name()='mobyData'
and @*[local-name()='queryID']='id_01']/*[local-name()='Simple' and
@*[local-name()='articleName']='sequence']"
will give us a "Simple" with articleName = "sequence";
Best regards,
Dmitry
From martin.senger at gmail.com Sat Jul 26 07:52:30 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Sat, 26 Jul 2008 19:52:30 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <488B0123.3040608@bsc.es>
References:
<488B0123.3040608@bsc.es>
Message-ID: <4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
> WSRF has...
Sorry, if you wish to use WSRF - I am out. Happy hacking...
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From martin.senger at gmail.com Sun Jul 27 21:29:34 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Mon, 28 Jul 2008 09:29:34 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
References:
<488B0123.3040608@bsc.es>
<4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
Message-ID: <4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
I am trying to figure out how I would implement the data by reference in
order to achieve the main purpose - not no have all data in memory. I know
how I would do it for the references being on the primitive types level, but
it is less clear to me how I would do it with the references on other
levels. Perhaps you can help me to explain how you would do it (or even
Pieter can explore how he is already doing it)?
For references on the primitive type level, I would do a usual parsing of a
Moby message and when I find a reference, I would resolve it do a local file
and I would pass to my service class the data as a file reference. It would
be up to the service to read the file contents to the memory or not - that
would depend what it needs to do with the data.
However, on the higher level, such as mobyData level, the reference must be
treated differently. If I do the same (resolving the reference to a local
file), I still need to call a parser again to parse the contents of the
local file. Because I still want to give my service already parsed data (and
not a biomoby XML), and because I do not want to have all in memory, I need
to create local files for each primitive type and do the same as above
(passing my service local file references). I do not see much other choices.
Am I missing a point here?
The above is doable, of course. But you see why I wanted to have references
only on the primitive type level. If we have them on any level (as it seems
to be) I need, in my implementation, actually make another (local)
references for the primitive type, anyway. Of course, it is better than the
remote references because I do not need to make a network connection for
each individual primitive type (point taken, Pieter) - but it still to be
done.
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From gordonp at ucalgary.ca Sun Jul 27 21:50:30 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Sun, 27 Jul 2008 19:50:30 -0600
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
References: <488B0123.3040608@bsc.es> <4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
<4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
Message-ID: <488D25E6.7030601@ucalgary.ca>
Hi Martin,
> I am trying to figure out how I would implement the data by reference in
> order to achieve the main purpose - not no have all data in memory.
I think you are coming at this from a different side than most of us:
data-by-reference is primarily to reduce the bandwidth burden by not
requiring results to go back to the client before being sent to another
service.
Paul
From markw at illuminae.com Sun Jul 27 23:58:56 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Sun, 27 Jul 2008 20:58:56 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <488D25E6.7030601@ucalgary.ca>
References:
<488B0123.3040608@bsc.es>
<4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
<4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
<488D25E6.7030601@ucalgary.ca>
Message-ID:
On Sun, 27 Jul 2008 18:50:30 -0700, Paul Gordon
wrote:
> Hi Martin,
>> I am trying to figure out how I would implement the data by reference in
>> order to achieve the main purpose - not no have all data in memory.
> I think you are coming at this from a different side than most of us:
> data-by-reference is primarily to reduce the bandwidth burden by not
> requiring results to go back to the client before being sent to another
> service.
I don't think that is the PRIMARY reason, necessarily - we also want to
get around passing large objects in the SOAP message due to limitations in
the (Perl) SOAP libraries, and increasingly in the non-SOAP services where
passing large data inside of an XML tag doesn't allow for nice download
optimizations/caching, etc.
M
From gordonp at ucalgary.ca Mon Jul 28 00:03:21 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Sun, 27 Jul 2008 22:03:21 -0600
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References: <488B0123.3040608@bsc.es> <4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com> <4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com> <488D25E6.7030601@ucalgary.ca>
Message-ID: <488D4509.6070407@ucalgary.ca>
Perhaps we should make a statement of intendion before hammering out the
details then ? ;-)
Mark Wilkinson wrote:
> On Sun, 27 Jul 2008 18:50:30 -0700, Paul Gordon
> wrote:
>
>> Hi Martin,
>>> I am trying to figure out how I would implement the data by
>>> reference in
>>> order to achieve the main purpose - not no have all data in memory.
>> I think you are coming at this from a different side than most of us:
>> data-by-reference is primarily to reduce the bandwidth burden by not
>> requiring results to go back to the client before being sent to
>> another service.
>
> I don't think that is the PRIMARY reason, necessarily - we also want
> to get around passing large objects in the SOAP message due to
> limitations in the (Perl) SOAP libraries, and increasingly in the
> non-SOAP services where passing large data inside of an XML tag
> doesn't allow for nice download optimizations/caching, etc.
>
> M
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> !DSPAM:60005,488d415d1095014969098!
>
>
>
From gordonp at ucalgary.ca Mon Jul 28 00:07:14 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Sun, 27 Jul 2008 22:07:14 -0600
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <488D4509.6070407@ucalgary.ca>
References: <488B0123.3040608@bsc.es> <4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com> <4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com> <488D25E6.7030601@ucalgary.ca>
<488D4509.6070407@ucalgary.ca>
Message-ID: <488D45F2.40900@ucalgary.ca>
Okay, this is a very long thread, so I forgot the original message,
which indeed has both of these uses (except I don't limit my vision of
inter-service data passing to well-known entities as Martin gave in his
example).
Paul Gordon wrote:
> Perhaps we should make a statement of intendion before hammering out
> the details then ? ;-)
>
> Mark Wilkinson wrote:
>> On Sun, 27 Jul 2008 18:50:30 -0700, Paul Gordon
>> wrote:
>>
>>> Hi Martin,
>>>> I am trying to figure out how I would implement the data by
>>>> reference in
>>>> order to achieve the main purpose - not no have all data in memory.
>>> I think you are coming at this from a different side than most of
>>> us: data-by-reference is primarily to reduce the bandwidth burden by
>>> not requiring results to go back to the client before being sent to
>>> another service.
>>
>> I don't think that is the PRIMARY reason, necessarily - we also want
>> to get around passing large objects in the SOAP message due to
>> limitations in the (Perl) SOAP libraries, and increasingly in the
>> non-SOAP services where passing large data inside of an XML tag
>> doesn't allow for nice download optimizations/caching, etc.
>>
>> M
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>>
>>
>>
>>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> !DSPAM:60005,488d4268109502155522951!
>
>
>
From martin.senger at gmail.com Mon Jul 28 00:10:05 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Mon, 28 Jul 2008 12:10:05 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <488D4509.6070407@ucalgary.ca>
References:
<488B0123.3040608@bsc.es>
<4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
<4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
<488D25E6.7030601@ucalgary.ca>
<488D4509.6070407@ucalgary.ca>
Message-ID: <4d93f07c0807272110k66b54250q9373799210f28f22@mail.gmail.com>
> Perhaps we should make a statement of intendion before hammering out the
> details then ? ;-)
The first email in this thread says it (I believe):
Well, the first purpose (A) is obvious: we want to be able to deliver huge
> data from a service. So the service returns only a reference instead of the
> real data and the client can fetch the real data using some memory-friendly
> protocol (usualy a simple HTTP/GET).
>
> But it appeeared that it was not the only purpose. The second purpose (B)
> is to be able to send around already existing references (such as URLs of
> the EMBL or NCBI records).
>
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From soiland-reyes at cs.manchester.ac.uk Mon Jul 28 05:32:42 2008
From: soiland-reyes at cs.manchester.ac.uk (Stian Soiland-Reyes)
Date: Mon, 28 Jul 2008 10:32:42 +0100
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<4889cf38.1aba720a.1b32.3a36@mx.google.com>
Message-ID:
Sorry if you got this email a couple of times, I ended up getting
bounces from moby-l@ even if I was posting to moby-dev@ - and changed
the wrong subscription!
On Mon, Jul 28, 2008 at 10:23 AM, Stian Soiland-Reyes
wrote:
> On Fri, Jul 25, 2008 at 2:03 PM, Edward Kawas wrote:
>> Don't worry about Taverna! Whatever we decide will most likely be the same
>> amount of work. So let's just get it right.
>
> Taverna 2 has built-in support for references (and their
> dereferencing) - so you could implement any referencing scheme you
> like, really, there's just a very simple interface to implement:
>
> http://taverna.cvs.sourceforge.net/taverna/t2reference/t2reference-api/src/main/java/net/sf/taverna/t2/reference/ExternalReferenceSPI.java?view=markup
>
>
> Example implementation:
>
> http://taverna.cvs.sourceforge.net/taverna/t2reference/t2reference-core-extensions/src/main/java/net/sf/taverna/t2/reference/impl/external/http/HttpReference.java?view=markup
>
>
> Plain HTTP references should work out of the box, though. You could
> make a version of this and add your own added-value semantics to the
> URIs - and it could still also work with other non-biomoby-services
> that recognised HTTP-URLs.
>
> t2 allows you to add translators between different reference schemes -
> so if you ended up with say "urn:biomoby:fish/1337" and you knew that
> you could download such references by using
> http://download.biomoby.org/fish/1337.xml - you can easily add a
> BiomobyReferenceToHttpReference translator that constructs such URIs
> given the biomoby reference.
>
>
> --
> Stian Soiland-Reyes, myGrid team
> School of Computer Science
> The University of Manchester
> http://www.mygrid.org.uk/dev/blog/
>
--
Stian Soiland-Reyes, myGrid team
School of Computer Science
The University of Manchester
http://www.mygrid.org.uk/dev/blog/
From jmfernandez at cnio.es Mon Jul 28 12:25:29 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Mon, 28 Jul 2008 18:25:29 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es> <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com> <488884CD.9030403@cnio.es> <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com> <2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com> <4889B5E8.6010903@mpiz-koeln.mpg.de> <4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
Message-ID: <488DF2F9.5090207@cnio.es>
Last weeks we (at INB) were talking about binary objects and data-by-reference
in our internal list (mainly related to the usage of XInclude), and we found
that on one hand XInclude allows referencing non-XML content:
http://www.w3.org/TR/xinclude/#text-included-items
but that content must be string-like (with an associated encoding), not binary
one. So, when we realized that limitation we were talking about extending
XInclude in a non-standard way, considering a new parse style, called
'base64'. Standard XInclude understands parse='xml' and parse='text' styles,
and 'base64' parse mode would work like 'text' one, but encoding the content
in base64 before including it in the place of the reference.
What do you think about this hack?
Best Regards,
Jos? Mar?a
Mark Wilkinson wrote:
> On Fri, 25 Jul 2008 05:28:43 -0700, Pieter Neerincx
> wrote:
>
>
>
>> If you want to replace this with a reference the actual data would
>> still have to be a base64 encoded piece of text. So you can not make a
>> reference to an image that exists as a plain JPEG, GIF, etc. on a web
>> server.
>
> Martin and I discussed this while he was here and the behaviour of this
> case will be part of his RFC proposal. It seems to me that it would be
> silly to b64 encode something that is being passed by reference... but
> I agree that to do anything else is a 'hack'. Nevertheless, there are a
> lot of 'hacks' in Moby, and this one is one of the more sensible 'hacks'
> IMO...
>
> M
>
>
>
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.
From py at pingoured.fr Tue Jul 29 05:09:26 2008
From: py at pingoured.fr (Pierre-Yves Chibon)
Date: Tue, 29 Jul 2008 11:09:26 +0200
Subject: [MOBY-dev] php client for biomoby services
Message-ID: <488EDE46.5070804@pingoured.fr>
Dear list,
We have been trying to make a php client for a simple biomoby web
service that we developed few days ago.
We are facing a problem in the creation of the input XML. We think that
using php soap library we create soap XML and not moby XML. Which
obviously returns an error.
Has anyone of you ever try to make a simple client in php or just even
call a bioMoby service via php ?
Below is the input that is sent to our service and the XML retrieve from
that service
Thanks for your help,
Best regards,
Pierre
input:
a:1:{s:12:"Solstis_User";a:3:{s:8:"userName";s:4:"test";s:8:"password";s:4:"test";s:5:"email";s:12:"test at test.fr";}}
output:
<?xml version="1.0"
encoding="UTF-8"?>
<moby:MOBY xmlns:moby="http://www.biomoby.org/moby">
<moby:mobyContent>
<moby:serviceNotes>
<moby:mobyException moby:severity="error">
<moby:exceptionCode>600</moby:exceptionCode>
<moby:exceptionMessage>AN ERROR OCCURED DURING THE
SERVICE EXECUTION: Error in the XML input.
SystemID: null
Line (perhaps): 1
Column: 1</moby:exceptionMessage>
</moby:mobyException>
</moby:serviceNotes>
</moby:mobyContent>
</moby:MOBY>
From edward.kawas at gmail.com Tue Jul 29 10:56:04 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Tue, 29 Jul 2008 07:56:04 -0700
Subject: [MOBY-dev] php client for biomoby services
In-Reply-To: <488EDE46.5070804@pingoured.fr>
References: <488EDE46.5070804@pingoured.fr>
Message-ID: <488f2f88.2a528c0a.7d99.54ae@mx.google.com>
Hi Pierre,
Take a look at the following example that someone created at a TIGR
workshop.
https://biomoby.tigr.org/wiki/index.php/JCVI_Workshop#::_Step_5_-_how_to_cre
ate_a_MOBY_client_in_PHP_::
This example did work at the time and may still do the trick.
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Pierre-Yves Chibon
Sent: July-29-08 2:09 AM
To: MOBY-dev at lists.open-bio.org
Subject: [MOBY-dev] php client for biomoby services
Dear list,
We have been trying to make a php client for a simple biomoby web
service that we developed few days ago.
We are facing a problem in the creation of the input XML. We think that
using php soap library we create soap XML and not moby XML. Which
obviously returns an error.
Has anyone of you ever try to make a simple client in php or just even
call a bioMoby service via php ?
Below is the input that is sent to our service and the XML retrieve from
that service
Thanks for your help,
Best regards,
Pierre
input:
a:1:{s:12:"Solstis_User";a:3:{s:8:"userName";s:4:"test
";s:8:"password";s:4:"test";s:5:"email";s:12:"test at test.fr";}}
output:
<?xml version="1.0"
encoding="UTF-8"?>
<moby:MOBY xmlns:moby="http://www.biomoby.org/moby">
<moby:mobyContent>
<moby:serviceNotes>
<moby:mobyException moby:severity="error">
<moby:exceptionCode>600</moby:exceptionCode>
<moby:exceptionMessage>AN ERROR OCCURED DURING THE
SERVICE EXECUTION: Error in the XML input.
SystemID: null
Line (perhaps): 1
Column: 1</moby:exceptionMessage>
</moby:mobyException>
</moby:serviceNotes>
</moby:mobyContent>
</moby:MOBY>
<
/soapenv:Envelope>
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From edward.kawas at gmail.com Tue Jul 29 11:23:58 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Tue, 29 Jul 2008 08:23:58 -0700
Subject: [MOBY-dev] php client for biomoby services
In-Reply-To: <488EDE46.5070804@pingoured.fr>
References: <488EDE46.5070804@pingoured.fr>
Message-ID: <488f3612.20d7720a.6190.1e4a@mx.google.com>
Just in case, you didn't get it working (I am sure that you did, but just in
case ...), I attached 2 php files that you can test with.
The test client is in the file 'moby_php_test_service.php' and the
moby_soap_factory.php needs to be either in the same directory as the test
service or on your PHP include path.
Thanks to Julian Tonti-Filippini for creating the PHP basis for the attached
examples!
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Pierre-Yves Chibon
Sent: July-29-08 2:09 AM
To: MOBY-dev at lists.open-bio.org
Subject: [MOBY-dev] php client for biomoby services
Dear list,
We have been trying to make a php client for a simple biomoby web
service that we developed few days ago.
We are facing a problem in the creation of the input XML. We think that
using php soap library we create soap XML and not moby XML. Which
obviously returns an error.
Has anyone of you ever try to make a simple client in php or just even
call a bioMoby service via php ?
Below is the input that is sent to our service and the XML retrieve from
that service
Thanks for your help,
Best regards,
Pierre
input:
a:1:{s:12:"Solstis_User";a:3:{s:8:"userName";s:4:"test
";s:8:"password";s:4:"test";s:5:"email";s:12:"test at test.fr";}}
output:
<?xml version="1.0"
encoding="UTF-8"?>
<moby:MOBY xmlns:moby="http://www.biomoby.org/moby">
<moby:mobyContent>
<moby:serviceNotes>
<moby:mobyException moby:severity="error">
<moby:exceptionCode>600</moby:exceptionCode>
<moby:exceptionMessage>AN ERROR OCCURED DURING THE
SERVICE EXECUTION: Error in the XML input.
SystemID: null
Line (perhaps): 1
Column: 1</moby:exceptionMessage>
</moby:mobyException>
</moby:serviceNotes>
</moby:mobyContent>
</moby:MOBY>
<
/soapenv:Envelope>
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moby_php_test_service.php
Type: application/octet-stream
Size: 1308 bytes
Desc: not available
URL:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moby_soap_factory.php
Type: application/octet-stream
Size: 4300 bytes
Desc: not available
URL:
From groscurt at mpiz-koeln.mpg.de Wed Jul 30 04:51:58 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 30 Jul 2008 10:51:58 +0200
Subject: [MOBY-dev] [MOBY-l] Apologies,
a hint for MIPS-Service substitutions, and an invitation
In-Reply-To:
References:
Message-ID: <48902BAE.7000501@mpiz-koeln.mpg.de>
Mark wrote:
> This all brings me to my final comment - I currently have two
> summer-students who are working on the problem of automated workflow
> repair (i.e. finding equivalent services to fix broken workflows).
> We're going to be inviting everyone to participate in a Moby Service
> Annotation Jamboree later this summer. It will be a "virtual"
> jamboree - just come to the website. We'll have an interface there
> through which you can deeply annotate Moby services (inputs, outputs,
> underlying resources, service types, authorities, etc.) with the goal
> of improving service discovery. I encourage all Moby users to
> participate and improve the quality of our service annotations! At a
> minimum, I hope that the current service providers will visit the
> website and annotate their own services :-)
I was wondering how you store this information ? are they mapped to the
services/datatypes etc in the moby central database ?
Here at the mpiz we already thought about offering this feature as a
permanent service to everyone, so everyone can tag services/datatypes
etc. Have you ever considered such kind of things ? I recently modified
dashboard and my local moby database to have a prototype for service
tagging. Unfortunately i can only offer you some screenshots, but I was
wondering if such extension could be worth considering to integrate it
in the official moby api ? you can see the first idea at
http://bioinfo.mpiz-koeln.mpg.de/tagging2.jpg
so my suggestion is to think about a tagging system (via dashboard / web
portal), which is permant available and allows everyone to tag moby stuff ?!
best
andreas
--
=====================================================
Dipl. Bioinf. Andreas Groscurth
Bioinformatics Software Developer
Plant Computational Biology group
Max-Planck Institute for plant breeding research
Carl-von-Linne Weg 10
50829 Cologne
Germany
+49(0) 221 5062449
=====================================================
From pieter.neerincx at gmail.com Wed Jul 30 09:51:54 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Wed, 30 Jul 2008 15:51:54 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
References:
<488B0123.3040608@bsc.es>
<4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
<4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
Message-ID:
Hi Martin,
Had to think about the issues you raised below for a while...
On 28 Jul 2008, at 03:29, Martin Senger wrote:
> I am trying to figure out how I would implement the data by
> reference in
> order to achieve the main purpose - not no have all data in memory.
> I know
> how I would do it for the references being on the primitive types
> level, but
> it is less clear to me how I would do it with the references on other
> levels. Perhaps you can help me to explain how you would do it (or
> even
> Pieter can explore how he is already doing it)?
What I use mostly are http(s) references. I download them and save
them as tmp files in a tmp dir. What happens next completely depends
on the service.
Sometimes I'm lazy and simply use a DOM parser to load the entire
thing in memory for convenient access to all nodes. (Note that I'm
living in a Perl world and therefore this makes sense. Using a
reference prevents DOM parsing of the SOAP message by SOAP::Lite which
has an expansion factor 5-7 times worse than DOM parsing of only the
Moby payload.)
If I cannot afford to be lazy, because the Moby payload tends to be
big, I use hybrid parsers. I never use pure streaming (SAX) parsers.
Hybrid means I use a streaming parser to divide the input into certain
chunks/fragments. Next I loop over the chunks and load them completely
into memory. This way I only have one chunk at a time in memory plus a
few globals like for example a counter to keep track of the amount of
BLAST hits. This hybrid parsing can be either for XML or for legacy
data formats like BLAST reports in tabular format.
In one case I use an XSLT to parse and convert one type of Moby XML
message into an SVG (is also XML) image. The XSLT is pretty cool and
is also some kind of hybrid. Basically it works similar to a flat file
indexing system. You define certain nodes of interest. It parses the
XML (SAX not DOM) once to make an index of where these nodes occur in
the XML. This index is kept in memory, but that is a lot smaller since
it stores only where a certain node occurs not it's contents and it
stores only nodes you need and specified not everything. Next it
parses the XML again (SAX not DOM) to do your business logic.
The inconvenience of a pure streaming parser is that you cannot jump
around the data. Once the streaming has parsed and passed a certain
point it forgets about that data unless you decide to store it in
variables. So, if you parse 100 BLAST hits and when you parse number
88 you want to go back to compare it to number 12, you can not unless
you stored number 12 somehow. With the XSLT you don't have to store
it, as you could have indexed the BLAST hits. So, when you need 12
again the parser jumps back to that point and parses only the chunk/
fragment for number 12 again. This means some parts of the XML might
get parsed more than once, but for scalability that is much better
than parsing the entire XML several times or storing large pieces in
memory.
The disadvantage of XLSTs in my opinion is that it works radically
different than any language I've seen before. Perl and Java are
different, but they share many concepts like if, else, while, for
loops. Developing XSLTs requires a completely different way of
thinking and I never found it easy.
> For references on the primitive type level, I would do a usual
> parsing of a
> Moby message and when I find a reference, I would resolve it do a
> local file
> and I would pass to my service class the data as a file reference.
> It would
> be up to the service to read the file contents to the memory or not
> - that
> would depend what it needs to do with the data.
>
> However, on the higher level, such as mobyData level, the reference
> must be
> treated differently. If I do the same (resolving the reference to a
> local
> file), I still need to call a parser again to parse the contents of
> the
> local file. Because I still want to give my service already parsed
> data (and
> not a biomoby XML), and because I do not want to have all in memory,
> I need
> to create local files for each primitive type and do the same as above
> (passing my service local file references). I do not see much other
> choices.
Not only would you have to do that for the primitives. If you want to
provide pre-parsed data to a service you would have to do it for the
values of namespace and id attributes at any level of mobyData too
right? I assume we are talking about MoSeS here were based on the data
in BioMoby Central code is generated both to parse the Biomoby XML
inputs of a service as well as to compile its results as BioMoby XML.
Saving everything pre-parsed as tmp files to disk, doesn't make much
sense too me. If you would do it both for the entire message and for
all it's possible dissected parts it would cause quite some redundancy
and overhead.
But how to do it with a hybrid DOM/SAX parser or with an XSLT that
uses indices? You would have to generate the code to chunk the input
for a hybrid parser or you have to generate the code to index certain
nodes for the XSLT. This means you need to know how the business logic
of the services works to figure out at which level level of the XML
you would have to chunk or index. I know Mark has this famous slide
with the God of BioMoby, but I'm afraid even the almighty Martin
cannot predict based on the info in BioMoby Central how the business
logic of a services works :(... In the current MoSeS parsing of the
input and the business logic are two separate things, but if you want
to improve scalability with a hybrid SAX/DOM or XSLT parser, parsing
the input will become an essential part of the business logic. What
you could do is generate some disabled example code to chunk/index at
any level and then a developer could uncomment the lines required.
Furthermore there are some obvious levels at which chunking/indexing
makes sense. Take for example a huge Collection of Simples. It would
make sense to chunk/index at the level of the pseudo-root element of
the Simples. I don't think it's possible to pre-generate all code to
parse the input in the right order and with chunking/indexing at the
right levels, but with a few smart examples a developer should be able
to quickly change the order and/or adapt to chunking/indexing further
up or down the tree.
> Am I missing a point here?
No, making a typical "currency convertor" example web service with a
single small input and a single small output is easy, but making a web
service that scales well is really hard!
There is a Perl module in the BioMoby CVS for XSLT based parsing. It
was developed in France some time ago if I remember correctly and
AFAIK it is not maintained, so it probably lacks some of the newer
stuff like proper error handling, async services etc., but it might be
a nice example of what can be done with XSLTs.
Ok, those of you who managed to read all the way up till here earned a
beer on the next BioMoby developers meeting!
Cheers,
Pi
> The above is doable, of course. But you see why I wanted to have
> references
> only on the primitive type level. If we have them on any level (as
> it seems
> to be) I need, in my implementation, actually make another (local)
> references for the primitive type, anyway. Of course, it is better
> than the
> remote references because I do not need to make a network connection
> for
> each individual primitive type (point taken, Pieter) - but it still
> to be
> done.
>
> Cheers,
> Martin
>
> --
> Martin Senger
> email: martin.senger at gmail.com,m.senger at cgiar.org
> skype: martinsenger
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
-------------------------------------------------------------
From markw at illuminae.com Wed Jul 30 10:54:23 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Wed, 30 Jul 2008 07:54:23 -0700
Subject: [MOBY-dev] [MOBY-l] Apologies,
a hint for MIPS-Service substitutions, and an invitation
In-Reply-To: <48902BAE.7000501@mpiz-koeln.mpg.de>
References:
<48902BAE.7000501@mpiz-koeln.mpg.de>
Message-ID:
On Wed, 30 Jul 2008 01:51:58 -0700, Andreas Groscurth
wrote:
> I was wondering how you store this information ? are they mapped to the
> services/datatypes etc in the moby central database ?
At the moment we're not entirely sure where to put it. I am loathe to put
it in Moby Central as we don't yet know what the quality of the annotation
will be (though I guess Moby Central isnt' curated either, so... it
probably wouldn't hurt). Moreover, the Moby Central API isn't able to
query at the level of detail that we will be capturing these annotations,
so there's no compelling reason to put it there. For now, we're putting
it in a local RDF store, but we may eventually move it into Freebase
(http://www.freebase.com/) to enable direct queries there. we'll see how
it goes.
> so my suggestion is to think about a tagging system (via dashboard / web
> portal), which is permant available and allows everyone to tag moby
> stuff ?!
Certainly I think we need a dashboard panel that contains the annotation
interface (called "ED" - the Entity Describer http://entitydescriber.org;
written by our very own Eddie Kawas :-) ). ED is currently a bit too
tightly linked to Connotea, but we're in the process of ripping it out and
making it more standalone for this Jamboree.
Eddie, what do you think of a dashboard ED panel?
M
--
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada
From edward.kawas at gmail.com Wed Jul 30 10:59:10 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 30 Jul 2008 07:59:10 -0700
Subject: [MOBY-dev] [MOBY-l] Apologies,
a hint for MIPS-Service substitutions, and an invitation
In-Reply-To:
References: <48902BAE.7000501@mpiz-koeln.mpg.de>
Message-ID: <489081c6.22bd720a.2434.ffffe4ae@mx.google.com>
I like the dashboard panel idea a lot. Seems like the best place to keep
that information. Nice thinking Andreas!
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Mark Wilkinson
Sent: July-30-08 7:54 AM
To: Core developer announcements; moby-l at lists.open-bio.org
Subject: Re: [MOBY-dev] [MOBY-l] Apologies, a hint for MIPS-Service
substitutions, and an invitation
On Wed, 30 Jul 2008 01:51:58 -0700, Andreas Groscurth
wrote:
> I was wondering how you store this information ? are they mapped to the
> services/datatypes etc in the moby central database ?
At the moment we're not entirely sure where to put it. I am loathe to put
it in Moby Central as we don't yet know what the quality of the annotation
will be (though I guess Moby Central isnt' curated either, so... it
probably wouldn't hurt). Moreover, the Moby Central API isn't able to
query at the level of detail that we will be capturing these annotations,
so there's no compelling reason to put it there. For now, we're putting
it in a local RDF store, but we may eventually move it into Freebase
(http://www.freebase.com/) to enable direct queries there. we'll see how
it goes.
> so my suggestion is to think about a tagging system (via dashboard / web
> portal), which is permant available and allows everyone to tag moby
> stuff ?!
Certainly I think we need a dashboard panel that contains the annotation
interface (called "ED" - the Entity Describer http://entitydescriber.org;
written by our very own Eddie Kawas :-) ). ED is currently a bit too
tightly linked to Connotea, but we're in the process of ripping it out and
making it more standalone for this Jamboree.
Eddie, what do you think of a dashboard ED panel?
M
--
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From groscurt at mpiz-koeln.mpg.de Thu Jul 31 03:32:28 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Thu, 31 Jul 2008 09:32:28 +0200
Subject: [MOBY-dev] [MOBY-l] Apologies,
a hint for MIPS-Service substitutions, and an invitation
In-Reply-To:
References: <48902BAE.7000501@mpiz-koeln.mpg.de>
Message-ID: <48916A8C.2090801@mpiz-koeln.mpg.de>
Mark Wilkinson wrote:
>
>> I was wondering how you store this information ? are they mapped to
>> the services/datatypes etc in the moby central database ?
>
> At the moment we're not entirely sure where to put it. I am loathe to
> put it in Moby Central as we don't yet know what the quality of the
> annotation will be
Considering some registered namespaces and especially datatypes -
annotations will be probably of low quality, but that we have already at
some point and it will always be with uncurated community systems.
> Moreover, the Moby Central API isn't able to query at the level of
> detail that we will be capturing these annotations, so there's no
> compelling reason to put it there.
yeah the central api and the client api has to be changed then...
although the searching for services via tags is not difficult on client
side.
>> so my suggestion is to think about a tagging system (via dashboard /
>> web portal), which is permant available and allows everyone to tag
>> moby stuff ?!
>
> Certainly I think we need a dashboard panel that contains the
> annotation interface (called "ED" - the Entity Describer
> http://entitydescriber.org; written by our very own Eddie Kawas :-)
> ). ED is currently a bit too tightly linked to Connotea, but we're in
> the process of ripping it out and making it more standalone for this
> Jamboree.
>
> Eddie, what do you think of a dashboard ED panel?
looks like a good idea for it ... nice to see something already
'somehow' existing :)
Best
andreas
--
=====================================================
Dipl. Bioinf. Andreas Groscurth
Bioinformatics Software Developer
Plant Computational Biology group
Max-Planck Institute for plant breeding research
Carl-von-Linne Weg 10
50829 Cologne
Germany
+49(0) 221 5062449
=====================================================
From dmitry.repchevski at bsc.es Thu Jul 31 05:10:08 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 31 Jul 2008 11:10:08 +0200
Subject: [MOBY-dev] Binary data streaming
Message-ID: <48918170.2060300@bsc.es>
Hello everybody,
Last INB meeting in Valencia we were agreed the the correct way to
implement streaming would be using XOP/MTOM specifications.
Even there is no any concrete document or agreement I was played with it
and IMO the easiest way to introduce it in Moby is through a new
primitive type,
something like "Binary". I implemented it in my package and it's very
easy to do using JAXB.
The schema for such an element is:
*****************************************
... moby attributes
*****************************************
Such a binary type will be encoded as base64 if there is no MTOM support
and as a MIME attachment (streamed) otherwise.
Here are both examples of encoding (with and without XOP/MTOM) support:
*****************************************
...
dGVzdA==
...
*****************************************
--uuid:0895f30d-7b88-411f-94a8-172a5187c9f7
Content-Id:
Content-Type: application/xop+xml;charset=utf-8;type="text/xml"
Content-Transfer-Encoding: binary
...
...
--uuid:0895f30d-7b88-411f-94a8-172a5187c9f7
Content-Id:
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
test
--uuid:0895f30d-7b88-411f-94a8-172a5187c9f7--
*****************************************
This way a PERL implementation (which could face a problem with MTOM
implementation) would use the first form and a Java one, the second...
Looking forward for comments,
Dmitry
From martin.senger at gmail.com Thu Jul 31 20:03:03 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Fri, 1 Aug 2008 08:03:03 +0800
Subject: [MOBY-dev] Binary data streaming
In-Reply-To: <48918170.2060300@bsc.es>
References: <48918170.2060300@bsc.es>
Message-ID: <4d93f07c0807311703t5a361e9fmb4e44f8bb2663e87@mail.gmail.com>
> the correct way to implement streaming
What do you mean by "implement streaming"? I concluded that you meant
"sending binary data from (or to) a moby service". Is is correct?
> would be using XOP/MTOM specifications
This is again a SOAP-based specification, meaning another way how to make an
attachment (correct?).
I have feeling that our current proposal about the data-by-reference (that I
am going to summarize in the next few hours) has a reasonable tools for
getting binary data - if sent as a reference. If the protocol for
dereferencing is HTTP, we can use MIME type (content-type HTTP header) for
references data.
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From pieter.neerincx at gmail.com Thu Jul 31 21:45:33 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Fri, 1 Aug 2008 03:45:33 +0200
Subject: [MOBY-dev] Binary data streaming
In-Reply-To: <4d93f07c0807311703t5a361e9fmb4e44f8bb2663e87@mail.gmail.com>
References: <48918170.2060300@bsc.es>
<4d93f07c0807311703t5a361e9fmb4e44f8bb2663e87@mail.gmail.com>
Message-ID:
Hi,
On 01 Aug 2008, at 02:03, Martin Senger wrote:
>> the correct way to implement streaming
>
>
> What do you mean by "implement streaming"? I concluded that you meant
> "sending binary data from (or to) a moby service". Is is correct?
>
>
>> would be using XOP/MTOM specifications
>
>
> This is again a SOAP-based specification, meaning another way how to
> make an
> attachment (correct?).
Well theoretically an attachment would be ideal. It makes sure the
attachment can be parsed separately to provide scalability.
Furthermore, you always keep the data "together". With "external"
references the data is separated and has to go over the wire in two
different sessions. This way data might get lost. If the first session
uses SOAP over HTTP with a reference to data available only over FTP
and my client or service doesn't talk FTP, I received only half of the
package. With an attachment it either arrives completely or it doesn't
arrive at all.
So, theoretically I'd love attachment support for BioMoby services,
but my previous questions remained unanswered:
1. Is there any Perl support for such XOP/MTOM attachments. I mean a
module that actually *works* in contrast to the MIME-tools package.
2. Do XOP/MTOM attachments scale beyond the reach of the casual
firewall? Most firewalls block MIME/DIME attachments over 5 MB in
size. Attachments ain't going to improve scalability of BioMoby
services if 99% of our users scale to a max of 5 MB.
Martin, if you are at the EBI right now, you might ask for some advise
at the external services group. I know they are one of the few smart
guys who managed to get their SOAP services working with MIME
attachments and hit the firewall limits. It was at ISMB in Fortaleza
2006 when I talked to them about these issues, so the situation might
have improved by now...
Still, even if we wouldn't need it for scalability, pass-by-reference
would be very handy so you can reference existing data from others
(like a reference to a UniProt record on a EBI server.)
So unless the issues above are resolved, attachments are theoretically
maybe a nicer solution, but I'd say pass-by-reference is practically
the best solution to improve scalability in a pragmatic way.
Cheers,
Pi
> I have feeling that our current proposal about the data-by-reference
> (that I
> am going to summarize in the next few hours) has a reasonable tools
> for
> getting binary data - if sent as a reference. If the protocol for
> dereferencing is HTTP, we can use MIME type (content-type HTTP
> header) for
> references data.
>
> Martin
>
> --
> Martin Senger
> email: martin.senger at gmail.com,m.senger at cgiar.org
> skype: martinsenger
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
-------------------------------------------------------------
From soiland-reyes at cs.manchester.ac.uk Mon Jul 28 05:23:47 2008
From: soiland-reyes at cs.manchester.ac.uk (Stian Soiland-Reyes)
Date: Mon, 28 Jul 2008 10:23:47 +0100
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4889cf38.1aba720a.1b32.3a36@mx.google.com>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<4889cf38.1aba720a.1b32.3a36@mx.google.com>
Message-ID:
On Fri, Jul 25, 2008 at 2:03 PM, Edward Kawas wrote:
> Don't worry about Taverna! Whatever we decide will most likely be the same
> amount of work. So let's just get it right.
Taverna 2 has built-in support for references (and their
dereferencing) - so you could implement any referencing scheme you
like, really, there's just a very simple interface to implement:
http://taverna.cvs.sourceforge.net/taverna/t2reference/t2reference-api/src/main/java/net/sf/taverna/t2/reference/ExternalReferenceSPI.java?view=markup
Example implementation:
http://taverna.cvs.sourceforge.net/taverna/t2reference/t2reference-core-extensions/src/main/java/net/sf/taverna/t2/reference/impl/external/http/HttpReference.java?view=markup
Plain HTTP references should work out of the box, though. You could
make a version of this and add your own added-value semantics to the
URIs - and it could still also work with other non-biomoby-services
that recognised HTTP-URLs.
t2 allows you to add translators between different reference schemes -
so if you ended up with say "urn:biomoby:fish/1337" and you knew that
you could download such references by using
http://download.biomoby.org/fish/1337.xml - you can easily add a
BiomobyReferenceToHttpReference translator that constructs such URIs
given the biomoby reference.
--
Stian Soiland-Reyes, myGrid team
School of Computer Science
The University of Manchester
http://www.mygrid.org.uk/dev/blog/
From martin.senger at gmail.com Tue Jul 1 18:41:49 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 1 Jul 2008 11:41:49 -0700
Subject: [MOBY-dev] data by reference - a request for comments
Message-ID: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
Hi all,
Yesterday, Mark, Eddie and I, we spent some time to evaluate what was
proposed during our last meeting about sending data by references. Here are
some thoughts that may crystalize into a request fro comments.
*What is the purpose of sending data by reference
*
Well, the first purpose (A) is obvious: we want to be able to deliver huge
data from a service. So the service returns only a reference instead of the
real data and the client can fetch the real data using some memory-friendly
protocol (usualy a simple HTTP/GET).
But it appeeared that it was not the only purpose. The second purpose (B) is
to be able to send around already existing references (such as URLs of the
EMBL or NCBI records). The existence of this B purpose makes the problem a
bit harder but it is a valid puspose. So the first question is:
*Do you agree that we pursue both purposes in this requests?
*
*The machinery
*
a) A service claims (in the registration time) that it can provide data by
reference.
b) A client asks for getting back references by including "acceptRefs"
attribute in mobyData tag. The attribute lists one or more protocol names
that the client can accept.
c) A service *can* obey such request and send one or more *primitive
data*as references (the focus on primitive type is new, originaly we
thought
about allowing references on any level, but now, mainly becuse of the
purpose B we do not propose it anymore). It can use any of the protocols
mentioned in the client's "acceptRefs" attribute. It can send references
only if at least one protocol matches.
*How does a client knows what protocols a service supports?*
This is a fundamental question that goes closely with "use existing
standards rather than inventing your own". An ideal solution is perhaps
this: A service returns not a reference to data itself but a reference to a
WSDL document that contains all supported protocols, including the endpoints
for this particular data. It is a nice idea but it breaks the purpose B - we
cannot use existing references without wrapping them first in a WSDL
document. The WSDL is strong because it gives us actually an *interface* how
to get data, but it is weak because the references cannot be used as *
indexes* (e.g. for further caching). Also, it does not solve the client
side: the "acceptRefs" attribute still needs to use a list of protocol names
(and not a WSD document because clients cannot make WSDL documents visible
to the world).
After going there and back, we concluded (and it is now our proposal for the
request of comments) that the service returns a reference to data, and
clients can deduce what protocol to us by looking at the protocol part of
the returned URL. We are aware that this is fine for usual protocol, such as
HTTP and FTP, but it cannot serve data, for example, by a SOAP. But, as
Eddie pointed out, if somebody wants SOAP for data, she can return data
directly in the Moby message.
*The remaining questions
*
Dmitry suggested to use WSRF. We think that he meant something else: It
could be used instead of the whole Moby message - but that is not what we
are looking for. We are looking for replacing just data part by references,
and we want still to keep the original Moby message as it is used now. So we
have concluded: no WSRF.
*How can a client tells a service that she is sending a reference instead of
data?* This could be useful for chaining services. We have not talked about
it. Ideas welcome.
The machinery described above may not allow to find, in advance, what
protocols a service is able to provide. It depends on what a service can
register into a moby central registry. It can be just a boolen flag ("I can
provid references"), or a list of supported protocols ("names"), or actually
nothing. The latest option has an advantage that no change in the registry
is needed. *Can we live with this simple option?*
I am not sure if I covered all, but better to send ti now and wait for your
comments.
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From jmfernandez at cnio.es Tue Jul 1 19:04:40 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Tue, 01 Jul 2008 21:04:40 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
Message-ID: <486A7FC8.20201@cnio.es>
Hi everybody,
I have added inline some personal comments about the RFC:
Martin Senger wrote:
> Hi all,
>
> Yesterday, Mark, Eddie and I, we spent some time to evaluate what was
> proposed during our last meeting about sending data by references. Here are
> some thoughts that may crystalize into a request fro comments.
>
> *What is the purpose of sending data by reference
> *
> Well, the first purpose (A) is obvious: we want to be able to deliver huge
> data from a service. So the service returns only a reference instead of the
> real data and the client can fetch the real data using some memory-friendly
> protocol (usualy a simple HTTP/GET).
>
> But it appeeared that it was not the only purpose. The second purpose (B) is
> to be able to send around already existing references (such as URLs of the
> EMBL or NCBI records). The existence of this B purpose makes the problem a
> bit harder but it is a valid puspose. So the first question is:
>
> *Do you agree that we pursue both purposes in this requests?
> *
> *The machinery
> *
> a) A service claims (in the registration time) that it can provide data by
> reference.
>
It can be claimed from the RDF of the service as a predicate, including the
understood reference protocols (which should be described elsewhere in an RDF
ontology).
> b) A client asks for getting back references by including "acceptRefs"
> attribute in mobyData tag. The attribute lists one or more protocol names
> that the client can accept.
>
I suggest using serviceNotes child element (called for instance acceptRefs
:-), telling what it can understand) instead of adding a new attribute to
mobyData, in both ways of the MOBY dialog. The client uses the acceptRefs
element in the message sent to the service to tell what it can understand, and
the service answers using the acceptRefs explaining what it have done. The
service must answer a subset of the understood reference protocols to the
client in order to maintain the contract.
If the service cannot fulfill the request where client does not understand the
protocols' the service is able to handle (because it refuses to send too much
information inline), it can fail using a new exception code related to message
size.
If the service uses references which are not understood by the client, then
the client can mark the message or the service as corrupted or offensive.
> c) A service *can* obey such request and send one or more *primitive
> data*as references (the focus on primitive type is new, originaly we
> thought
> about allowing references on any level, but now, mainly becuse of the
> purpose B we do not propose it anymore). It can use any of the protocols
> mentioned in the client's "acceptRefs" attribute. It can send references
> only if at least one protocol matches.
>
> *How does a client knows what protocols a service supports?*
>
> This is a fundamental question that goes closely with "use existing
> standards rather than inventing your own". An ideal solution is perhaps
> this: A service returns not a reference to data itself but a reference to a
> WSDL document that contains all supported protocols, including the endpoints
> for this particular data. It is a nice idea but it breaks the purpose B - we
> cannot use existing references without wrapping them first in a WSDL
> document. The WSDL is strong because it gives us actually an *interface* how
> to get data, but it is weak because the references cannot be used as *
> indexes* (e.g. for further caching). Also, it does not solve the client
> side: the "acceptRefs" attribute still needs to use a list of protocol names
> (and not a WSD document because clients cannot make WSDL documents visible
> to the world).
>
A passive, static way is publishing the service capabilities about references
in the RDF. An active, dynamic way is creating a new service port (like we did
with asynchronous services), which provides live metadata information about
the service. I could even be linked/related to LSID metadata servers, but it
is still outside my knowledge.
> After going there and back, we concluded (and it is now our proposal for the
> request of comments) that the service returns a reference to data, and
> clients can deduce what protocol to us by looking at the protocol part of
> the returned URL. We are aware that this is fine for usual protocol, such as
> HTTP and FTP, but it cannot serve data, for example, by a SOAP. But, as
> Eddie pointed out, if somebody wants SOAP for data, she can return data
> directly in the Moby message.
>
> *The remaining questions
> *
> Dmitry suggested to use WSRF. We think that he meant something else: It
> could be used instead of the whole Moby message - but that is not what we
> are looking for. We are looking for replacing just data part by references,
> and we want still to keep the original Moby message as it is used now. So we
> have concluded: no WSRF.
>
> *How can a client tells a service that she is sending a reference instead of
> data?* This could be useful for chaining services. We have not talked about
> it. Ideas welcome.
>
> The machinery described above may not allow to find, in advance, what
> protocols a service is able to provide. It depends on what a service can
> register into a moby central registry. It can be just a boolen flag ("I can
> provid references"), or a list of supported protocols ("names"), or actually
> nothing. The latest option has an advantage that no change in the registry
> is needed. *Can we live with this simple option?*
>
> I am not sure if I covered all, but better to send ti now and wait for your
> comments.
>
> Cheers,
> Martin
>
Best Regards,
Jos? Mar?a
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.
From martin.senger at gmail.com Wed Jul 2 05:22:30 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 1 Jul 2008 22:22:30 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486A7FC8.20201@cnio.es>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
<486A7FC8.20201@cnio.es>
Message-ID: <4d93f07c0807012222i1ae4f97ai152796660a0bc46a@mail.gmail.com>
> It can be claimed from the RDF of the service as a predicate, including the
> understood reference protocols (which should be described elsewhere in an
> RDF ontology).
I do not like using the RDF for mandatory features. The RDF, at least in the
current moby, is optional and should not carry anything that is significant
for the service and its behaviour. The data by reference should be part of
the main BioMoby API (or whatever we call it).
> I suggest using serviceNotes child element
I do not have a strong opinion where in the message this could be - I have
just used the tag name and location that was suggested in our February
meeting. The ServiceNotes, however, seems a bit a strange place when used by
the clients.
> and the service answers using the acceptRefs explaining what it have done.
Why? Why is not enough the reference itself? It is a URL and the URL has a
wel-defined protocol. And what if a service uses a different protocol for
different parts of the message (remember the purpose B)? No, the
ServiceNotes is not a suitable place for it.
> The service must answer a subset of the understood reference protocols to
> the client in order to maintain the contract.
And what about if the service does not want to return references, at all?
That should be an error, as well? No, I don't think so. A client just
suggests that she is able to accept references but there is no guarantee
that she can get it (many services will not have any provision for
references).
> A passive, static way is publishing the service capabilities about
> references in the RDF.
It can be there, why not, - but not as the main way.
> An active, dynamic way is creating a new service port (like we did with
> asynchronous services)
Do you mean a new method name? But for what? Not sure I understand how this
new method may help here (and not breaking the purpose B, for example).
Just my opinions.
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From dmitry.repchevski at bsc.es Wed Jul 2 10:26:52 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Wed, 02 Jul 2008 12:26:52 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
Message-ID: <486B57EC.7090302@bsc.es>
Hello!
Here are my comments. Please note that all they are in mode IMHO,
because I understand the complicity of the problem and just looking for
possible solution.
> [A] Well, the first purpose (A) is obvious: we want to be able to deliver huge data from a service.
It depends.
If we are going to stay with SOAP (and especially move to doc/lit) the
right way is to provide streaming through SOAP Attachments (SwA) or even
better using MTOM.
This way we have to introduce a special Moby Primitive Type something
called "BinaryAttachment" which will be bind to [type="base64Binary"]
This type will be replaced with:
...
--MIME_boundary
Content Type:image/gif
Content-Transfer-Encoding:binary
Content-ID:
... (file content)
--MIME_boudary
In case of using MTOM:
...
------=_Part_1_4558657.1118953559446
Content-Type: image/jpeg
Content-ID:
Obviously JAVA supports both, but considering Perl, I am not sure which
is better
(I am very pessimistic in SOAP Perl capabilities... :-( )
> The second purpose (B) is to be able to send around already existing references (such as URLs of the EMBL or NCBI records).
Here is the question where ever we continue to invent standards or
trying to use SOAP capabilities...
Finally we have WS-Addressing (and even have it implemented in Perl :-) ).
More we have WSRF that is exactly for this purpose - reference for resource.
> Dmitry suggested to use WSRF. We think that he meant something else: It
> could be used instead of the whole Moby message - but that is not what we
> are looking for.
I am afraid you didn't understand me well... WSRF doesn't mean we have
to send ALL the message.
It is only the way to reference a resource. Even in moby-asynchrony we
return the whole message if needn't be so.
We can perfectly return a reference to ANY part of the original request.
It is up to the service to locate the asked resource and send it back to
the WSRF request.
We shouldn't confuse "A" and "B" because they are different.
"A" is to stream the data which in many case means "give me a huge file".
"B" is to keep the reference to it (something temporal like a
calculation that produces a lot of data and we don't want to pass it
around).
> We are looking for replacing just data part by references,
> and we want still to keep the original Moby message as it is used now. So we
> have concluded: no WSRF.
Don't understand it... How you suppose to replace some object in Moby
message by refernce without changing it?
> *How can a client tells a service that she is sending a reference instead of
> data?* This could be useful for chaining services. We have not talked about
> it. Ideas welcome.
There is no way to specify that it can be send both by value and by
reference and the same type...
I suggest - the service expecting the usual "value", but check if there
is WSRF resource sent in header.
If it is it just calls the service (or in case it is the same server,
just locate it locally) to get the object (not all Moby message, but
just serialized Moby object, like AminoacidSequence) and use it instead
the EMPTY one in MobyMessage.
Something like - We expect two AminoacidSequence(s) -
articleName="first", articleName="second"
In header we put WS Addressing (WSRF) resource to "server1/second"
The service check headers for resources and call the server1 to obtain
"second" Aminoacid.
This can be useful to make a workflow too... because we can call the
server1 to get AminoacidSequence2, obtain reference to it AND send it to
the server 2, so actually server2 will get the reference to it...
In case server1 and the server2 are the same (this is the main idea of
[B]?) we do not need to pass around and get it directly... in case of
server1 != server2 we have a ... Moby "GRID" :D
> I do not like using the RDF for mandatory features. The RDF, at least in the
> current moby, is optional and should not carry anything that is significant
> for the service and its behaviour.
Agree. IMHO I would use WSDL capability to describe it all, but having
so many types of services this could be very hard...
IMO the main problem is that BioMoby is using SOAP, but use nothing of
it trying to replace all W3C and WS-I specification by it's own solution.
So the question is why do we use SOAP?
I see (in a long term) two possibilities:
1. Forget about SOAP and do the REST (which is a fancy name to say we
are going to send what ever we want without any standard)
2. Use SOAP and Web Services architecture.
Best regards,
Dmitry
From markw at illuminae.com Wed Jul 2 14:22:13 2008
From: markw at illuminae.com (Mark)
Date: Wed, 02 Jul 2008 07:22:13 -0700
Subject: [MOBY-dev] Interesting...
Message-ID:
http://www.biomedcentral.com/1471-2105/9/295/abstract
I wonder how they got document-literal stuff working in Perl??
M
From markw at illuminae.com Wed Jul 2 14:11:29 2008
From: markw at illuminae.com (Mark)
Date: Wed, 02 Jul 2008 07:11:29 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486B57EC.7090302@bsc.es>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
<486B57EC.7090302@bsc.es>
Message-ID:
>> The second purpose (B) is to be able to send around already existing
>> references (such as URLs of the EMBL or NCBI records).
> Here is the question where ever we continue to invent standards or
> trying to use SOAP capabilities...
> Finally we have WS-Addressing (and even have it implemented in Perl :-)
> ).
> More we have WSRF that is exactly for this purpose - reference for
> resource.
Hi Dmitry!
Could you send an example of a Moby message that has multiple components
(e.g., the image and annotation component of SimpleAnnotatedJpegImage)
where the content of the data-carrying primitives are references using the
WS-Addressing standard? that way, we'll all know without any ambiguity
what it is you are suggesting.
Thanks!
Mark
From dmitry.repchevski at bsc.es Wed Jul 2 17:56:51 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Wed, 02 Jul 2008 19:56:51 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
Message-ID: <486BC163.9050007@bsc.es>
Hello Mark,
My suggestion (not a final specification :-)) is something like:
******************************
http://myserver.com/MyService?asyncId=ID
ID
mobyws:result_queryId00/image/jpeg/content
description
***********************
The idea is to include the WSRF reference into the header:
***********************
http://myserver.com/MyService?asyncId=ID
ID
mobyws:result_queryId00/image/jpeg/content
***********************
The service can access the header looking for resources and then to call
WSRF service specified to obtain them.
Of course we must update our WSRF stack to return a particular moby
object instead of all mobyData. here I "updated" the like to
"mobyws:result_queryId00/image/jpeg/content"
Note that I used a simplest possible way to produce an... uri (?) to it
- using "articleNames". I commented the "content" in the body to
demonstrate the idea better - it is up to service implementation to call
a wsrf and obtain the image... This way we do not need to change "moby"
message.
Cheers,
Dmitry
From gordonp at ucalgary.ca Wed Jul 2 17:57:40 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Wed, 02 Jul 2008 11:57:40 -0600
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807012222i1ae4f97ai152796660a0bc46a@mail.gmail.com>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com> <486A7FC8.20201@cnio.es>
<4d93f07c0807012222i1ae4f97ai152796660a0bc46a@mail.gmail.com>
Message-ID: <486BC194.90506@ucalgary.ca>
Hi Martin,
I agree with most of your post, but I will be pedantic and point out the
trap you, and previously Tom Oinn, fell into in this discussion:
> Why? Why is not enough the reference itself? It is a URL and the URL has a
> wel-defined protocol. ...
URL does not in fact have a well-defined protocol. URL is simply a way
to express "protocol:protocol-specific-stuff"
I'll assume you mean http and maybe ftp protocols. It's important to be
specific, because you probably don't want to include mailto:, gopher: or
prospero: for example.
Paul
From markw at illuminae.com Wed Jul 2 18:21:57 2008
From: markw at illuminae.com (Mark)
Date: Wed, 02 Jul 2008 11:21:57 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486BC163.9050007@bsc.es>
References: <486BC163.9050007@bsc.es>
Message-ID:
Do you suggest that there is no indication, in the MOBY block, that the
message contains data-by-reference? i.e. should our tools simply see that
the primitive is empty, and then look into the ws header for the
possibility of a ws-ref, and then if it isn't found, adopt the other
interpretation of an empty block (i.e. that the data is known to be NULL)
?
M
On Wed, 02 Jul 2008 10:56:51 -0700, Dmitry Repchevsky
wrote:
> Hello Mark,
>
> My suggestion (not a final specification :-)) is something like:
>
> ******************************
>
>
>
>
> xmlns:wsa="http://www.w3.org/2005/08/addressing">
> http://myserver.com/MyService?asyncId=ID
>
>
> ID
>
>
>
> mobyws:result_queryId00/image/jpeg/content
>
>
>
>
>
>
>
>
>
>
>
> articleName="Description">description
>
>
>
>
>
>
>
>
> ***********************
>
> The idea is to include the WSRF reference into the header:
>
> ***********************
> xmlns:wsa="http://www.w3.org/2005/08/addressing">
> http://myserver.com/MyService?asyncId=ID
>
> xmlns:mobyws="http://biomoby.org/">ID
>
>
> mobyws:result_queryId00/image/jpeg/content
>
>
> ***********************
>
> The service can access the header looking for resources and then to call
> WSRF service specified to obtain them.
> Of course we must update our WSRF stack to return a particular moby
> object instead of all mobyData. here I "updated" the like to
> "mobyws:result_queryId00/image/jpeg/content"
> Note that I used a simplest possible way to produce an... uri (?) to it
> - using "articleNames". I commented the "content" in the body to
> demonstrate the idea better - it is up to service implementation to call
> a wsrf and obtain the image... This way we do not need to change "moby"
> message.
>
> Cheers,
>
> Dmitry
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
--
--
Mark Wilkinson
Heart + Lung Research Institute
Centre for High Throughput Biology
UBC
From dmitry.repchevski at bsc.es Wed Jul 2 22:15:13 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 03 Jul 2008 00:15:13 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
Message-ID: <486BFDF1.1070001@bsc.es>
Hello,
No I think that the service (tool?) should look into the header first
and if there are some references, ignore corresponding element from the
body. As I said this is only an idea. This way we do not change the
message itself and do can implement it on the base of existing wsrf code...
As of the wsrf response for
//"mobyws:result_queryId00/image/jpeg/content"// the service will return
the standard MOBY message but only with elements found in the path
(MOBY/mobyContent/mobyData/Simple/SimpleAnnotatedJPEGImage/String
Best regards,
Dmitry
From edward.kawas at gmail.com Wed Jul 2 22:33:07 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 2 Jul 2008 15:33:07 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486BFDF1.1070001@bsc.es>
References: <486BFDF1.1070001@bsc.es>
Message-ID: <486c0227.07ec720a.5dc1.ffffd76a@mx.google.com>
With all this talk about WSRF, SOAP, etc, what would happen with cgi (i.e.
non moby/moby-async) services?
I think that the solution needs to be able to encompass all categories of
services that moby claims to support and not just [JAVA] SOAP services.
Just a few of my thoughts,
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Dmitry Repchevsky
Sent: July-02-08 3:15 PM
To: moby-dev at lists.open-bio.org
Subject: Re: [MOBY-dev] data by reference - a request for comments
Hello,
No I think that the service (tool?) should look into the header first
and if there are some references, ignore corresponding element from the
body. As I said this is only an idea. This way we do not change the
message itself and do can implement it on the base of existing wsrf code...
As of the wsrf response for
//"mobyws:result_queryId00/image/jpeg/content"// the service will return
the standard MOBY message but only with elements found in the path
(MOBY/mobyContent/mobyData/Simple/SimpleAnnotatedJPEGImage/String
Best regards,
Dmitry
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From martin.senger at gmail.com Thu Jul 3 04:11:26 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Wed, 2 Jul 2008 21:11:26 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486BC194.90506@ucalgary.ca>
References: <4d93f07c0807011141y73f5f919g169a1c01d93d6c95@mail.gmail.com>
<486A7FC8.20201@cnio.es>
<4d93f07c0807012222i1ae4f97ai152796660a0bc46a@mail.gmail.com>
<486BC194.90506@ucalgary.ca>
Message-ID: <4d93f07c0807022111u4f65fe7ap851a997a23e230a8@mail.gmail.com>
Paul,
URL does not in fact have a well-defined protocol. URL is simply a way to
> express "protocol:protocol-specific-stuff"
Point taken. The URL has a well-defined *place* where I can find a protocol,
but it does not define what protocol can sit on that place.
> I'll assume you mean http and maybe ftp protocols. It's important to be
> specific, because you probably don't want to include mailto:, gopher: or
> prospero: for example.
>
Well, I did not mean to be too specific. All the protocols you mentioned
would be possible (but some of them may hardly find a match between a
service an a client).
All,
I am not against standards, actually standards was what I was hired to
promote at EBI, but my experiences are that sometimes standards simply are
an obstacle. Especially for BioMoby - that is anyway a non-standard message
protocol. We already burned ourselves with using WSRF for asynchronous
services (we have now problems how to make asynchronous services using
REST). I am afraid that using WSRF will rather move service provider from
Moby world than closer to it (I am talking from the point of view of the
service providers and the tools developers, not from the perspective of Mark
who has to face the funding agencies).
The fact is that we want to use REST (in the near future) and not only SOAP,
we want to use Perl where SOAP is not that well supported, and we want the
solution for data by reference to be simple.
Therefore, I do not wish to be eaten (again) by the WSRF - unless my boss
(whoever it is or will be) forces me to do so, of course.
BTW, I like very much the Dmitry's definition of the REST ("a fancy name to
say we are going to send what ever we want without any standard"). I agree
with him - except that we are sending Moby message - which is not standard,
it is faulty, but it has worked relatively fine so far.
For being able to continue this discussion (about data by reference), I
suggest that we try to answer first whether we want to use only SOAP or
whether we are really heading to use REST, as well. My vote for this
question is: yes, we will want to have the REST services, as well.
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From dmitry.repchevski at bsc.es Thu Jul 3 08:46:40 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 03 Jul 2008 10:46:40 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486c0227.07ec720a.5dc1.ffffd76a@mx.google.com>
References: <486c0227.07ec720a.5dc1.ffffd76a@mx.google.com>
Message-ID: <486C91F0.4000100@bsc.es>
>
> With all this talk about WSRF, SOAP, etc, what would happen with cgi (i.e.
> non moby/moby-async) services?
Interesting question... How do we implement all possible functionality
using all possible solutions... :-)
Here is my point of view:
1. "moby" service
It can work with "wsrf" pass by reference... it's up to the toolkit
to look into the header
Attachment (SwA). Just put your attachment outside the soap message
and find it somehow from the toolkit
2 "moby-async" service
The same as "moby", but also can return a reference (through wsrf) to
a particular moby object
3. "doc-literal" service
The same as "moby", but attachment would be MTOM (backward compatible
with SwA).
4. "doc-literal-async"
Also see no problem.
5. "soap", "cgi", "wsdl", "post" - what's this??? (sorry for ingnorance)
Best regards,
Dmitry
From machiel.jansen at sara.nl Thu Jul 3 09:13:14 2008
From: machiel.jansen at sara.nl (Machiel Jansen)
Date: Thu, 3 Jul 2008 11:13:14 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486C91F0.4000100@bsc.es>
References: <486c0227.07ec720a.5dc1.ffffd76a@mx.google.com>
<486C91F0.4000100@bsc.es>
Message-ID:
For all I know WSRF is already been surpassed by WS-RT. http://www.ibm.com/developerworks/grid/library/gr-wsrfwsrt/index.html?ca=drs-
There are quite a number of people who are disenchanted by WSRF. See http://blog.harbulot.com/post/2006/11/22/Experiences-with-WSRF for one example.
Machiel
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org [mailto:moby-dev-
> bounces at lists.open-bio.org] On Behalf Of Dmitry Repchevsky
> Sent: donderdag 3 juli 2008 10:47
> To: moby-dev at lists.open-bio.org
> Subject: Re: [MOBY-dev] data by reference - a request for comments
>
> >
> > With all this talk about WSRF, SOAP, etc, what would happen with cgi
> (i.e.
> > non moby/moby-async) services?
> Interesting question... How do we implement all possible functionality
> using all possible solutions... :-)
>
> Here is my point of view:
>
> 1. "moby" service
> It can work with "wsrf" pass by reference... it's up to the toolkit
> to look into the header
> Attachment (SwA). Just put your attachment outside the soap message
> and find it somehow from the toolkit
>
> 2 "moby-async" service
> The same as "moby", but also can return a reference (through wsrf)
> to
> a particular moby object
>
> 3. "doc-literal" service
> The same as "moby", but attachment would be MTOM (backward
> compatible
> with SwA).
>
> 4. "doc-literal-async"
> Also see no problem.
>
>
> 5. "soap", "cgi", "wsdl", "post" - what's this??? (sorry for ingnorance)
>
> Best regards,
>
> Dmitry
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
From dmitry.repchevski at bsc.es Thu Jul 3 09:20:32 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 03 Jul 2008 11:20:32 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807022111u4f65fe7ap851a997a23e230a8@mail.gmail.com>
References: <4d93f07c0807022111u4f65fe7ap851a997a23e230a8@mail.gmail.com>
Message-ID: <486C99E0.4040402@bsc.es>
Hello Martin,
So as far as I understand, the final battle is about Rest/Soap... So
let's use CORBA!!! (joke)
I am not the one that like all the standards (especially the
latest_and_greatest ones), but as a developer I recognize that we can
not compete against the whole industry. My usual position is to use
those standards that have a strong support in a Java world (of course I
am Java biased). So I see what we have from w3c, an then to JCP... I
don't want to implement them by myself... I am lazy...
With WSRF I had no other way, but implement a part of it. Here we have a
problem, because in distributed computations (and Moby is a kind of it)
Grid is very popular and so WSRF, but somehow Sun guys ignore it...
I understand SOAP and how it works... We can do all the things we need
just USING it.
Even current Moby message format can fit into it with XML Schema. My
moby implementation library doesn't parse moby message, but rather let
the standard Java library do it.
I am afraid we can not mix SOAP-WSRF approach with REST... Just because
the REST is not a technology, neither a protocol.
It's just a concept (and... IT IS STATELESS), so you can define your
protocol, but you will need to reinvent all the protocols... when a
resource should be destroyed? It reminds me the year 2000 when having
CORBA that worked just fine, someone said "It is complicated, let's
invent something simple"... Now we have SOAP with 4 different encodings
+ some 20 specifications + WSDL +... And all they do the same CORBA did
10 years ago...
Now people don't want SOAP because it is complicated so let's start
REST... OK, but I'd like to see the specification with
asynchrony/streaming based on REST concept.
What is why I love Java - There is no any Java specification without a
working implementation.
So, finally IMHO if we want to move out of SOAP we at least need to know
where we are moving...
Cheers,
Dmitry
From dmitry.repchevski at bsc.es Thu Jul 3 09:57:59 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 03 Jul 2008 11:57:59 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
Message-ID: <486CA2A7.4070702@bsc.es>
Hello,
> For all I know WSRF is already been surpassed by WS-RT. http://www.ibm.com/developerworks/grid/library/gr-wsrfwsrt/index.html?ca=drs-
> There are quite a number of people who are disenchanted by WSRF. See http://blog.harbulot.com/post/2006/11/22/Experiences-with-WSRF for one example.
>
> Machiel
This is what I am talking about... Martin has a reason we have had
enough problems with WSRF... now we have a "better" one (WSRT)...
But (!) There is an implementation? It looks like that this is a perfect
technology that exists in a paper...
Should we implement all the standards just to be able to use it? If IBM
claims to have a new/cool standard (by the way "standard" - who is
behind it?) but why can't I find (free) implementation of it?
It uses WS-Transfer has more sense to me because it can describe a way
to represent a moby object in moby message tree (for wsrf). and it is at
least w3s standard.
Claiming something to be a standard is always a bold claim...
Dmitry
From jflo at imim.es Tue Jul 8 11:42:04 2008
From: jflo at imim.es (Judith Flo)
Date: Tue, 08 Jul 2008 13:42:04 +0200
Subject: [MOBY-dev] strange behaviour
Message-ID: <4873528C.7070000@imim.es>
Hi,
Yesterday I changed one datatype (WeightedObject) to add another String
(Description). I unregistered the previous one and registered the new
one, same name, diferent structure.
Now the problem: when I try to compile the service, i get an error
telling me that the method that should set, this new attribute doesn't
exists. It is a very strange problem I've never seen before... The
curious is that Eclipse is not complaining and when I type object.set it
shows me as a possible operation to do, to set this atribute, but when I
try to compile it, just get:
[javac] Compiling 1 source file to
/home/judith/workspace3/services/WebContent/WEB-INF/classes
[javac]
/home/judith/workspace3/services/src/es/imim/cgl/runBlastXPImpl.java:76:
cannot find symbol
[javac] symbol : method set_Description(java.lang.String)
[javac] location: class org.biomoby.shared.datatypes.WeightedObject
[javac] w.set_Description("asdf");
[javac] ^
[javac] 1 error
Does someone know what's happening?
Of course I have regenerated the biomoby-datatypes.jar and now I check
that the new class exists and it contains this new atribute.
j
From jflo at imim.es Tue Jul 8 11:58:07 2008
From: jflo at imim.es (Judith Flo)
Date: Tue, 08 Jul 2008 13:58:07 +0200
Subject: [MOBY-dev] strange behaviour part 2
Message-ID: <4873564F.4020700@imim.es>
I also got this error on server side when trying to run some service
from dashboard:
AN ERROR OCCURED DURING THE SERVICE EXECUTION: Class
'org.biomoby.shared.datatypes.MapDataTypes' was not found.
It may indicate that you have not generated all Biomoby data types from
a Biomoby registry.
See http://www.biomoby.org/moby-live/Java/docs/Moses.html for details.
If you are a jMoby developer just type: ./build-dev.sh moses-datatypes.
Or perhaps, they just need to be compiled: ./build-dev.sh moses-compile.
From martin.senger at gmail.com Tue Jul 8 13:07:01 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 8 Jul 2008 08:07:01 -0500
Subject: [MOBY-dev] strange behaviour
In-Reply-To: <4873528C.7070000@imim.es>
References: <4873528C.7070000@imim.es>
Message-ID: <4d93f07c0807080607i402e659cy22532236854192d@mail.gmail.com>
> Yesterday I changed one datatype (WeightedObject) to add another String
> (Description). I unregistered the previous one and registered the new one,
> same name, diferent structure.
When changing an existing data type, the local cache should be updated. Or
newly created (reloading).
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From martin.senger at gmail.com Tue Jul 8 13:10:24 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 8 Jul 2008 08:10:24 -0500
Subject: [MOBY-dev] strange behaviour part 2
In-Reply-To: <4873564F.4020700@imim.es>
References: <4873564F.4020700@imim.es>
Message-ID: <4d93f07c0807080610p75acfcdaofc707d78abc1cb5a@mail.gmail.com>
> AN ERROR OCCURED DURING THE SERVICE EXECUTION: Class
> 'org.biomoby.shared.datatypes.MapDataTypes' was not found.
>
Look into your Tomcat's WEB-INF's lib directory and check what jar files you
have there, and if there are up-to-date.
> If you are a jMoby developer just type: ./build-dev.sh moses-datatypes.
>
Be aware that this message is a bit old (I have to update it in the code).
But you already know that you are using rather "ant moses-datatypes".
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From jflo at imim.es Tue Jul 8 14:51:14 2008
From: jflo at imim.es (Judith Flo)
Date: Tue, 08 Jul 2008 16:51:14 +0200
Subject: [MOBY-dev] strange behaviour part 2
In-Reply-To: <4d93f07c0807080610p75acfcdaofc707d78abc1cb5a@mail.gmail.com>
References: <4873564F.4020700@imim.es>
<4d93f07c0807080610p75acfcdaofc707d78abc1cb5a@mail.gmail.com>
Message-ID: <48737EE2.1010804@imim.es>
I've checked the lasts versions that are running (both in my machine and
the server) and the biomoby-datatypes.jar is the same (the last that I
generated with the object changed).
I did reload the cache before, I repeated now and it keeps complaining
about this new atribute/method.
I'll try to remove cache... maybe it'll help...
Any other idea?
j
Martin Senger wrote:
>> AN ERROR OCCURED DURING THE SERVICE EXECUTION: Class
>> 'org.biomoby.shared.datatypes.MapDataTypes' was not found.
>>
>>
>
> Look into your Tomcat's WEB-INF's lib directory and check what jar files you
> have there, and if there are up-to-date.
>
>
>
>> If you are a jMoby developer just type: ./build-dev.sh moses-datatypes.
>>
>>
>
> Be aware that this message is a bit old (I have to update it in the code).
> But you already know that you are using rather "ant moses-datatypes".
>
> Martin
>
>
From martin.senger at gmail.com Tue Jul 8 15:27:27 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 8 Jul 2008 10:27:27 -0500
Subject: [MOBY-dev] strange behaviour part 2
In-Reply-To: <48737EE2.1010804@imim.es>
References: <4873564F.4020700@imim.es>
<4d93f07c0807080610p75acfcdaofc707d78abc1cb5a@mail.gmail.com>
<48737EE2.1010804@imim.es>
Message-ID: <4d93f07c0807080827n1b63ef3yd4c8cfdc420983c6@mail.gmail.com>
> Any other idea?
Also restart yout Tomcat. Or, try your service "locally" (either from
Dashboard or from a command-line script) - if this works then the probelm is
in Tomcat.
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Mon Jul 14 06:52:10 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Mon, 14 Jul 2008 08:52:10 +0200
Subject: [MOBY-dev] Problem with MobyNamespace.getNamespace
Message-ID: <487AF79A.30301@mpiz-koeln.mpg.de>
Hi,
I'm using the method MobyNamespace.getNamespace(String ns) in the jmoby
api to retrieve the MobyNamespace for a given string. Unfortunately this
method is not working anymore correctly. E.g. if i use PMID as input
(which is a namespace) i get null back from this method.
I tested it with PDB, AFAWE_ID, PMID and UniProt, which all are namespaces.
The java code I'm using is some weeks old and it worked as far as i know
for a long long time - why not now ?
If i call http://biomoby.org/RESSOURCES/MOBY-S/Namespaces i get the rdf
in which I do find e.g. PMID... but still the method above returns null.
any ideas ?
thx
andreas
From gordonp at ucalgary.ca Mon Jul 14 13:57:02 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Mon, 14 Jul 2008 07:57:02 -0600
Subject: [MOBY-dev] Problem with MobyNamespace.getNamespace
In-Reply-To: <487AF79A.30301@mpiz-koeln.mpg.de>
References: <487AF79A.30301@mpiz-koeln.mpg.de>
Message-ID: <487B5B2E.4030104@ucalgary.ca>
Hi Andreas,
I'll check it out and get back to you...
Andreas Groscurth wrote:
> Hi,
>
> I'm using the method MobyNamespace.getNamespace(String ns) in the
> jmoby api to retrieve the MobyNamespace for a given string.
> Unfortunately this method is not working anymore correctly. E.g. if i
> use PMID as input (which is a namespace) i get null back from this
> method.
>
> I tested it with PDB, AFAWE_ID, PMID and UniProt, which all are
> namespaces.
>
> The java code I'm using is some weeks old and it worked as far as i
> know for a long long time - why not now ?
>
> If i call http://biomoby.org/RESSOURCES/MOBY-S/Namespaces i get the
> rdf in which I do find e.g. PMID... but still the method above returns
> null.
>
> any ideas ?
>
> thx
> andreas
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
>
From groscurt at mpiz-koeln.mpg.de Tue Jul 15 11:56:17 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Tue, 15 Jul 2008 13:56:17 +0200
Subject: [MOBY-dev] findService and the hierarchy
Message-ID: <487C9061.3030203@mpiz-koeln.mpg.de>
Hi,
another question from my side...
I use the findService method in jmoby in CentralImpl. Due to the code
all calls end up in the method findService(MobyService service, String[]
keyword, boolean includeChildrenServiceType, boolean
includeParentDatatype).
I use the MobyDatatype Annotated_GO_Term which is a child of GO_Term. In
my opinion when searching for services consuming Annotated_GO_Term via
the findService method i should also include the service which consumes
a GO_Term datatype, as this is the parent datatype (per default
'includeParentDatatype' is true)....
what is wrong in this way to think ? because i do not find service with
datatype GO_Term ? and if the way i think is wrong - can you give me an
example how this parameter is used at all ?
Thanks
andreas
From gordonp at ucalgary.ca Wed Jul 16 20:43:02 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Wed, 16 Jul 2008 14:43:02 -0600
Subject: [MOBY-dev] Problem with MobyNamespace.getNamespace
In-Reply-To: <487AF79A.30301@mpiz-koeln.mpg.de>
References: <487AF79A.30301@mpiz-koeln.mpg.de>
Message-ID: <487E5D56.8050508@ucalgary.ca>
Hi Andreas,
I've updated MobyNamespace to use Eddie's new caching mechanism, and it
seems to work again. You'll ned to do a CVS update for the whole the
moby-live/Java/src/main directory. Please let me know if you have any
more trouble. I will be updating MobyDataType and MobyServiceType
similarly.
Regards,
Paul
Andreas Groscurth wrote:
> Hi,
>
> I'm using the method MobyNamespace.getNamespace(String ns) in the
> jmoby api to retrieve the MobyNamespace for a given string.
> Unfortunately this method is not working anymore correctly. E.g. if i
> use PMID as input (which is a namespace) i get null back from this
> method.
>
> I tested it with PDB, AFAWE_ID, PMID and UniProt, which all are
> namespaces.
>
> The java code I'm using is some weeks old and it worked as far as i
> know for a long long time - why not now ?
>
> If i call http://biomoby.org/RESSOURCES/MOBY-S/Namespaces i get the
> rdf in which I do find e.g. PMID... but still the method above returns
> null.
>
> any ideas ?
>
> thx
> andreas
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
>
From groscurt at mpiz-koeln.mpg.de Fri Jul 18 08:00:34 2008
From: groscurt at mpiz-koeln.mpg.de (groscurt)
Date: Fri, 18 Jul 2008 10:00:34 +0200
Subject: [MOBY-dev] Problem with MobyNamespace.getNamespace
In-Reply-To: <487E5D56.8050508@ucalgary.ca>
Message-ID:
Hi Paul
thanks for your effort, it does work now :)
Best
andreas
----------------original message-----------------
From: "Paul Gordon" gordonp at ucalgary.ca
To: "Core developer announcements" moby-dev at lists.open-bio.org
Date: Wed, 16 Jul 2008 14:43:02 -0600
-------------------------------------------------
> Hi Andreas,
>
> I've updated MobyNamespace to use Eddie's new caching mechanism, and it
> seems to work again. You'll ned to do a CVS update for the whole the
> moby-live/Java/src/main directory. Please let me know if you have any
> more trouble. I will be updating MobyDataType and MobyServiceType
> similarly.
>
> Regards,
>
> Paul
>
> Andreas Groscurth wrote:
>> Hi,
>>
>> I'm using the method MobyNamespace.getNamespace(String ns) in the
>> jmoby api to retrieve the MobyNamespace for a given string.
>> Unfortunately this method is not working anymore correctly. E.g. if i
>> use PMID as input (which is a namespace) i get null back from this
>> method.
>>
>> I tested it with PDB, AFAWE_ID, PMID and UniProt, which all are
>> namespaces.
>>
>> The java code I'm using is some weeks old and it worked as far as i
>> know for a long long time - why not now ?
>>
>> If i call http://biomoby.org/RESSOURCES/MOBY-S/Namespaces i get the
>> rdf in which I do find e.g. PMID... but still the method above returns
>> null.
>>
>> any ideas ?
>>
>> thx
>> andreas
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
From martin.senger at gmail.com Wed Jul 23 03:09:48 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 22 Jul 2008 20:09:48 -0700
Subject: [MOBY-dev] findService and the hierarchy
In-Reply-To: <487C9061.3030203@mpiz-koeln.mpg.de>
References: <487C9061.3030203@mpiz-koeln.mpg.de>
Message-ID: <4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
> In my opinion when searching for services consuming Annotated_GO_Term via
> the findService method i should also include the service which consumes a
> GO_Term datatype, as this is the parent datatype (per default
> 'includeParentDatatype' is true)....
>
I assume that by "I should also include" you actually meant "the registry
response should also include". Is my understanding correct?
If so, I am afraid that it is not the issue of the jMoby library but the
registry itself. Eddie or Mark can confirm or reject this hypothesis.
The BioMoby API allows to specify two XML tags when calling findService
method: the
X
and
X
Where X can be either 1 or 0.
The jMoby's findService() method sets these two tags according to the two
boolean parameters that the method can get:
boolean includeChildrenServiceTypes
and
boolean includeParentDataTypes
I have tried to call the findService, with the Annotated_GO_Term input,
using all combinations of these two tags (you can do it using the Moby
command-line client with the optioins -nc and/or -np). The results are
either no service or 713 services:
expandObjects expandServices # of found services
1 1 713
0 0 0
1 0 713
0 0 0
I have feeling that this is not a correct result. Could Eddie or Mark
perhaps comment on it?
[I vaguely remember that we had this conversation about these tags with Mark
years(!) ago - but I do not recall what was the conclusion. I only, and
again vaguely, remember to have this issue in my to-do list - which would
indicate that something could be done in the jMoby library.]
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Wed Jul 23 08:05:48 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 23 Jul 2008 10:05:48 +0200
Subject: [MOBY-dev] findService and the hierarchy
In-Reply-To: <4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
References: <487C9061.3030203@mpiz-koeln.mpg.de>
<4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
Message-ID: <4886E65C.5060205@mpiz-koeln.mpg.de>
Yes Martin you understood it - as always - perfectly correct :)
Thanks for the additional information
Martin Senger wrote:
>> In my opinion when searching for services consuming Annotated_GO_Term via
>> the findService method i should also include the service which consumes a
>> GO_Term datatype, as this is the parent datatype (per default
>> 'includeParentDatatype' is true)....
>>
>>
>
> I assume that by "I should also include" you actually meant "the registry
> response should also include". Is my understanding correct?
>
> If so, I am afraid that it is not the issue of the jMoby library but the
> registry itself. Eddie or Mark can confirm or reject this hypothesis.
>
> The BioMoby API allows to specify two XML tags when calling findService
> method: the
>
> X
> and
> X
>
> Where X can be either 1 or 0.
>
> The jMoby's findService() method sets these two tags according to the two
> boolean parameters that the method can get:
>
> boolean includeChildrenServiceTypes
> and
> boolean includeParentDataTypes
>
> I have tried to call the findService, with the Annotated_GO_Term input,
> using all combinations of these two tags (you can do it using the Moby
> command-line client with the optioins -nc and/or -np). The results are
> either no service or 713 services:
>
> expandObjects expandServices # of found services
> 1 1 713
> 0 0 0
> 1 0 713
> 0 0 0
>
> I have feeling that this is not a correct result. Could Eddie or Mark
> perhaps comment on it?
>
> [I vaguely remember that we had this conversation about these tags with Mark
> years(!) ago - but I do not recall what was the conclusion. I only, and
> again vaguely, remember to have this issue in my to-do list - which would
> indicate that something could be done in the jMoby library.]
>
> Cheers,
> Martin
>
>
From groscurt at mpiz-koeln.mpg.de Wed Jul 23 08:54:56 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 23 Jul 2008 10:54:56 +0200
Subject: [MOBY-dev] working with images
In-Reply-To:
References:
Message-ID: <4886F1E0.5020704@mpiz-koeln.mpg.de>
Hi,
i try to figure out how to transfer images via Web Service in moby....
E.g. i use the b64_encoded_jpeg. It has the children 'content' which
requires a String, so i need to somehow encode/decode the image file so
that it can be transmitted as a String and the client can read it as an
image (which is somehow done via MobyDataBytes)...
how to do that ?
thanks
Andreas
From Sebastien.Carrere at toulouse.inra.fr Wed Jul 23 09:10:04 2008
From: Sebastien.Carrere at toulouse.inra.fr (Sebastien Carrere)
Date: Wed, 23 Jul 2008 11:10:04 +0200
Subject: [MOBY-dev] working with images
In-Reply-To: <4886F1E0.5020704@mpiz-koeln.mpg.de>
References:
<4886F1E0.5020704@mpiz-koeln.mpg.de>
Message-ID: <4886F56C.4090100@toulouse.inra.fr>
Hi Andreas,
In my web services (written using PlayMoby , then Perl), I use the
module MIME::Base64
(http://search.cpan.org/~gaas/MIME-Base64-3.07/Base64.pm).
I just make a cat on my picture file, and encode the result in Base64,
then add it to my Moby String content.
Sebastien
Andreas Groscurth wrote:
> Hi,
>
> i try to figure out how to transfer images via Web Service in moby....
>
> E.g. i use the b64_encoded_jpeg. It has the children 'content' which
> requires a String, so i need to somehow encode/decode the image file
> so that it can be transmitted as a String and the client can read it
> as an image (which is somehow done via MobyDataBytes)...
>
> how to do that ?
>
> thanks
> Andreas
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Sebastien_Carrere.vcf
Type: text/x-vcard
Size: 387 bytes
Desc: not available
URL:
From martin.senger at gmail.com Wed Jul 23 09:22:19 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Wed, 23 Jul 2008 17:22:19 +0800
Subject: [MOBY-dev] working with images
In-Reply-To: <4886F1E0.5020704@mpiz-koeln.mpg.de>
References:
<4886F1E0.5020704@mpiz-koeln.mpg.de>
Message-ID: <4d93f07c0807230222l1586b9f1r37b0c9f05f326c38@mail.gmail.com>
> i try to figure out how to transfer images via Web Service in moby....
You can look for an example in the
org.biomoby.service.dashboard.renderers.Base64Image.java. This example does
opposite thing that you want, but it is still the same. I used there the
library coming with Axis (which is used by BioMoby anyway, so it is not an
additional dependency):
import org.apache.axis.encoding.Base64;
The example uses method decode() - and you need to use encode() (I guess).
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Wed Jul 23 09:36:25 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 23 Jul 2008 11:36:25 +0200
Subject: [MOBY-dev] working with images
In-Reply-To: <4886F56C.4090100@toulouse.inra.fr>
References: <4886F1E0.5020704@mpiz-koeln.mpg.de>
<4886F56C.4090100@toulouse.inra.fr>
Message-ID: <4886FB99.1080404@mpiz-koeln.mpg.de>
Hi,
thanks for that - i forgot to mention i'm looking for a java solution ;-)
but your hint of encode in Base64 made me google again and found the
apache.commons.codec and it works.
Thanks for the hint :)
Andreas
Sebastien Carrere wrote:
> Hi Andreas,
>
> In my web services (written using PlayMoby , then Perl), I use the
> module MIME::Base64
> (http://search.cpan.org/~gaas/MIME-Base64-3.07/Base64.pm).
>
> I just make a cat on my picture file, and encode the result in Base64,
> then add it to my Moby String content.
>
>
> Sebastien
>
>
>
>
>
> Andreas Groscurth wrote:
>> Hi,
>>
>> i try to figure out how to transfer images via Web Service in moby....
>>
>> E.g. i use the b64_encoded_jpeg. It has the children 'content' which
>> requires a String, so i need to somehow encode/decode the image file
>> so that it can be transmitted as a String and the client can read it
>> as an image (which is somehow done via MobyDataBytes)...
>>
>> how to do that ?
>>
>> thanks
>> Andreas
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
From martin.senger at gmail.com Wed Jul 23 09:35:32 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Wed, 23 Jul 2008 17:35:32 +0800
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
Message-ID: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
I have deployed (to Maven) the current jMoby as version 1.1.0.
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From edward.kawas at gmail.com Wed Jul 23 12:54:08 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 23 Jul 2008 05:54:08 -0700
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
Message-ID: <488729f9.20f8720a.0551.0626@mx.google.com>
Hi Martin,
Does this update include org.biomoby.service.dashboard.* (the dashboard
components)?
Thanks,
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
Sent: July-23-08 2:36 AM
To: Core developer announcements
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
I have deployed (to Maven) the current jMoby as version 1.1.0.
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From martin.senger at gmail.com Wed Jul 23 13:04:10 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Wed, 23 Jul 2008 21:04:10 +0800
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <488729f9.20f8720a.0551.0626@mx.google.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
<488729f9.20f8720a.0551.0626@mx.google.com>
Message-ID: <4d93f07c0807230604p664530ag2dda536268aaf8a1@mail.gmail.com>
> Does this update include org.biomoby.service.dashboard.* (the dashboard
> components)?
I have not changed anything regarding included packages - my intention was
simply to have the latest code available from maven (because we need it for
some projects). I am aware that you want to have some separation of packages
- for that we may make another release sooner or later.
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From jmfernandez at cnio.es Wed Jul 23 13:04:02 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Wed, 23 Jul 2008 15:04:02 +0200
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <488729f9.20f8720a.0551.0626@mx.google.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
<488729f9.20f8720a.0551.0626@mx.google.com>
Message-ID: <48872C42.3060008@cnio.es>
Hi Eddie and Martin,
I have realized that there are two sets of jMoby artifacts. The first one
(biomoby.org maven group) is the used by taverna-biomoby plugin, meanwhile the
second one (org.biomoby maven group) is the officially provided by you.
So, which are the differences between them? The first one is already in
version 1.2, while the second one has just reached version 1.1.0.
Best Regards,
Jos? Mar?a
Edward Kawas wrote:
> Hi Martin,
>
> Does this update include org.biomoby.service.dashboard.* (the dashboard
> components)?
>
> Thanks,
>
> Eddie
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
> Sent: July-23-08 2:36 AM
> To: Core developer announcements
> Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
>
> I have deployed (to Maven) the current jMoby as version 1.1.0.
>
> Cheers,
> Martin
>
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.
From edward.kawas at gmail.com Wed Jul 23 13:13:19 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 23 Jul 2008 06:13:19 -0700
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <48872C42.3060008@cnio.es>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com> <488729f9.20f8720a.0551.0626@mx.google.com>
<48872C42.3060008@cnio.es>
Message-ID: <48872e78.16538c0a.696c.1560@mx.google.com>
Hi Jose Maria,
The version used by the taverna-biomoby plugin contains some extra classes.
Once these classes are available in the org.biomoby maven group, the
biomoby.org maven group will be wound down and the plugin will utilize the
official org.biomoby maven group.
Thanks,
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Jos? Mar?a
Fern?ndez Gonz?lez
Sent: July-23-08 6:04 AM
To: Core developer announcements
Subject: Re: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
Hi Eddie and Martin,
I have realized that there are two sets of jMoby artifacts. The
first one
(biomoby.org maven group) is the used by taverna-biomoby plugin, meanwhile
the
second one (org.biomoby maven group) is the officially provided by you.
So, which are the differences between them? The first one is already
in
version 1.2, while the second one has just reached version 1.1.0.
Best Regards,
Jos? Mar?a
Edward Kawas wrote:
> Hi Martin,
>
> Does this update include org.biomoby.service.dashboard.* (the dashboard
> components)?
>
> Thanks,
>
> Eddie
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
> Sent: July-23-08 2:36 AM
> To: Core developer announcements
> Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
>
> I have deployed (to Maven) the current jMoby as version 1.1.0.
>
> Cheers,
> Martin
>
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los
ficheros adjuntos, pueden contener informaci?n protegida para el uso
exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o
cualquier otro tipo de transmisi?n por parte de otra persona que no sea el
destinatario. Si usted recibe por error este correo, se ruega comunicarlo al
remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may
contain confidential and privileged information for the sole use of the
designated recipient named above. Distribution, reproduction or any other
use of this transmission by any party other than the intended recipient is
prohibited. If you are not the intended recipient please contact the sender
and delete all copies.
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From pieter.neerincx at gmail.com Wed Jul 23 15:06:38 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Wed, 23 Jul 2008 17:06:38 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <486CA2A7.4070702@bsc.es>
References:
<486CA2A7.4070702@bsc.es>
Message-ID: <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
Hi,
> Martin Senger wrote:
> A service *can* obey such request and send one or more *primitive
> data* as references (the focus on primitive type is new, originaly
> we thought about allowing references on any level, but now, mainly
> becuse of the purpose B we do not propose it anymore).
That doesn't sound very appealing to me. I use quite large data
structures and can have tens of thousands of them in a collection
(think of oligos for micro arrays.) If I can only replace the
primitives with references, I would still have to send a BioMoby XML
structure with hundreds of thousands of references replacing the
primitives. A client or the next service in a workflow would have to
resolve all those references with each reference pointing to only a
relatively small amount of data. That would be a mess and just cause a
huge amount of overhead.
So, I would really love to see the ability to replace a complete
collection or even the entire mobyData block with a reference. Having
said that I really like the idea of purpose "B" too, where I can embed
a link to existing EMBL/Genbank/etc. records on EBI/NCBI/etc. servers.
The only risk would be that a client or the next service in a workflow
happily resolves those links as fast and efficient as it can causing
too much load on those servers. A colleague once managed to have our
entire campus disconnected from the NCBI with simple script resolving
PubMed URLs... it was really easy :). With references it's always
possible to reference something on someone else's servers, but with
purpose "B" it's more likely to happen.
> Martin Senger wrote:
> I do not like using the RDF for mandatory features. The RDF, at
> least in the
> current moby, is optional and should not carry anything that is
> significant
> for the service and its behaviour. The data by reference should be
> part of
> the main BioMoby API (or whatever we call it).
I agree with Martin partially on this one: pass by reference should be
part of the main BioMoby API and not something optional as it is
essential for successful client-service communication. But that
doesn't necessarily exclude using RDF. We could also promote the RDF
stuff from optional to required. I don't want to open a can of worms
here and I'm not advocating making the RDF features mandatory. I'm
just saying it might be an option...
> Dmitry Repchevski wrote:
> If we are going to stay with SOAP (and especially move to doc/lit)
> the right way is to provide streaming through SOAP Attachments (SwA)
> or even better using MTOM.
I don't think attachments of any kind would be a good idea. Although
theoretically a nice solution for streaming data and keeping
everything together, there are practical problems:
* Most firewalls block attachments over 5 MB in size. One of the great
things of web services compared to the good 'ol CORBA is that we can
tunnel over any protocol. So most of us use HTTP(S) to bypass our
institute's paranoid sys admins and skip the part where you'd have to
bug them to open up another firewall port. I'm not sure if this
problem persists with the newer MTOM, but it's disaster with MIME or
DIME attachments.
* I'm not sure about the newer MTOM, but the combination of SOAP and
MIME attachments is a disaster of you're using Perl. The required
modules are dying projects and you'd have to patch them manually to
make it work. If we would go with attachments we would either drop
Perl support for BioMoby or one of us would have to write a SOAP::Lite
+ MIME::tools alternative...
I agree with Dmitry that if we are going to stick with a standard it
has to be a de facto standard. Hence well supported by toolkits/
implementations. A standard that only exists on paper is useless. Most
of us want to build or use BioMoby services and are not that much
interested in writing toolkits. I never managed to become a big WSRF
fan. It's needlessly complicated causing lot's of overhead.
Actually I like what Mark wrote initially most:
> We'll write-up the formal specification of how to do this soon, but
> briefly the idea is that we will use the xlink XML attribute in a
> Moby Object XML <.... xlink=''/>. When that tag is present, it is
> assumed that the content of that node is available at the URI in
> that reference. During service registration a provider will
> indicate the various transport protocols they provide for creating
> references (e.g. http, ftp) and this will be discoverable during a
> registry query. If you don't indicate a protocol, then you are
> saying that you do not support pass-by-reference, and therefore all
> existing services are supported. When accessing a service, you
> indicate to the service provider that you want data to be passed by
> reference by adding an attribute in the mobyData block acceptRefs="http ftp">. The service provider then has the option of
> providing you references for any data they wish. The xlink
> attribute can appear at any level in your Moby object, such that
> some data may be passed in the object itself, while other data from
> the same object may be passed by reference.
For the BioMoby data structures this only requires allowing an
optional xlink attribute and acceptRefs in the request from the
client. Maybe it's nicer to have acceptRefs as a child element of
mobyData or maybe at another level in the XML, but not as an
attribute, because attributes are intended for single items. In the
example above acceptRefs contains a space separated list and off
course we can parse that, but it's ugly XML. Anyway, what Mark wrote
above is very flexible, light-weight and relatively backwards
compatible. If I can choose between a simple <... xlink=""> attribute
or the bulky WSA header from Dmitry's example like in:
http://myserver.com/MyService?asyncId=ID
ID
mobyws:result_queryId00/image/jpeg/content
I'd choose the xlink attribute!
Cheers,
Pi
On 3?Jul?2008, at 11:57 AM, Dmitry Repchevsky wrote:
> Hello,
>
>> For all I know WSRF is already been surpassed by WS-RT. http://www.ibm.com/developerworks/grid/library/gr-wsrfwsrt/index.html?ca=drs-
>> There are quite a number of people who are disenchanted by WSRF.
>> See http://blog.harbulot.com/post/2006/11/22/Experiences-with-WSRF
>> for one example.
>>
>> Machiel
> This is what I am talking about... Martin has a reason we have had
> enough problems with WSRF... now we have a "better" one (WSRT)...
> But (!) There is an implementation? It looks like that this is a
> perfect technology that exists in a paper...
> Should we implement all the standards just to be able to use it? If
> IBM claims to have a new/cool standard (by the way "standard" - who
> is behind it?) but why can't I find (free) implementation of it?
>
> It uses WS-Transfer has more sense to me because it can describe a
> way to represent a moby object in moby message tree (for wsrf). and
> it is at least w3s standard.
> Claiming something to be a standard is always a bold claim...
>
> Dmitry
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
------------------------------------------------------------
From markw at illuminae.com Wed Jul 23 16:55:06 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Wed, 23 Jul 2008 09:55:06 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
Message-ID:
On Wed, 23 Jul 2008 08:06:38 -0700, Pieter Neerincx
wrote:
> That doesn't sound very appealing to me. I use quite large data
> structures and can have tens of thousands of them in a collection (think
> of oligos for micro arrays.) If I can only replace the primitives with
> references, I would still have to send a BioMoby XML structure with
> hundreds of thousands of references replacing the primitives.
Hmmm... that's a situation that we had NEVER considered! Putting xlinks
into a component of the Moby message higher than the mobyData block was
not a scenario that came up during our discussions in Tokyo. the xlink
was really only intended to carry data, not data + moby message
structure... but I suppose it COULD!
How do others feel about this?
M
--
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada
From jmfernandez at cnio.es Wed Jul 23 17:47:54 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Wed, 23 Jul 2008 19:47:54 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
Message-ID: <48876ECA.1040109@cnio.es>
IMHO it can be dangerous for the MOBY payload integrity using XInclude at
mobyContent childs (mobyData) level, but it can be needed in scenarios like
the one described by Pieter. For security reasons, XInclude / XLink / XPointer
should be restricted to the contents of mobyData, not to MOBY Data itself. For
instance, references to Simples or Collections, Collections composed by a
mixure of Simples and references to Simples, or Secondary parameters copied
from run to run.
Best Regards,
Jos? Mar?a
Mark Wilkinson wrote:
> On Wed, 23 Jul 2008 08:06:38 -0700, Pieter Neerincx
> wrote:
>
>
>> That doesn't sound very appealing to me. I use quite large data
>> structures and can have tens of thousands of them in a collection
>> (think of oligos for micro arrays.) If I can only replace the
>> primitives with references, I would still have to send a BioMoby XML
>> structure with hundreds of thousands of references replacing the
>> primitives.
>
> Hmmm... that's a situation that we had NEVER considered! Putting xlinks
> into a component of the Moby message higher than the mobyData block was
> not a scenario that came up during our discussions in Tokyo. the xlink
> was really only intended to carry data, not data + moby message
> structure... but I suppose it COULD!
>
> How do others feel about this?
>
> M
>
>
>
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.
From gordonp at ucalgary.ca Wed Jul 23 18:45:39 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Wed, 23 Jul 2008 12:45:39 -0600
Subject: [MOBY-dev] working with images
In-Reply-To: <4886FB99.1080404@mpiz-koeln.mpg.de>
References: <4886F1E0.5020704@mpiz-koeln.mpg.de> <4886F56C.4090100@toulouse.inra.fr>
<4886FB99.1080404@mpiz-koeln.mpg.de>
Message-ID: <48877C53.7010101@ucalgary.ca>
Hi Andreas,
If you're using the org.biomoby.shared.data package, there is a class
specifically for handling binary data:
http://biomoby.open-bio.org/CVS_CONTENT/moby-live/Java/docs/API/org/biomoby/shared/data/MobyDataBytes.html
Andreas Groscurth wrote:
> Hi,
>
> thanks for that - i forgot to mention i'm looking for a java solution ;-)
> but your hint of encode in Base64 made me google again and found the
> apache.commons.codec and it works.
>
> Thanks for the hint :)
> Andreas
>
> Sebastien Carrere wrote:
>> Hi Andreas,
>>
>> In my web services (written using PlayMoby , then Perl), I use the
>> module MIME::Base64
>> (http://search.cpan.org/~gaas/MIME-Base64-3.07/Base64.pm).
>>
>> I just make a cat on my picture file, and encode the result in
>> Base64, then add it to my Moby String content.
>>
>>
>> Sebastien
>>
>>
>>
>>
>>
>> Andreas Groscurth wrote:
>>> Hi,
>>>
>>> i try to figure out how to transfer images via Web Service in moby....
>>>
>>> E.g. i use the b64_encoded_jpeg. It has the children 'content' which
>>> requires a String, so i need to somehow encode/decode the image file
>>> so that it can be transmitted as a String and the client can read
>>> it as an image (which is somehow done via MobyDataBytes)...
>>>
>>> how to do that ?
>>>
>>> thanks
>>> Andreas
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> !DSPAM:60005,4886f8b9109503161834!
>
>
>
From pieter.neerincx at gmail.com Wed Jul 23 23:48:03 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Thu, 24 Jul 2008 01:48:03 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <48876ECA.1040109@cnio.es>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
Message-ID: <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
Hi Jos?,
I don't understand how references could be security risk. And if they
do why the level at which you could use them would impact security.
Could you please explain?
If references to mobyData blocks are not an option having at least
references to BioMoby articles would already be a big help.
Secondaries are usually very small so I wouldn't need / use references
there, but for primary Simple and especially for primary Collection
articles it would be a big plus!
Cheers,
Pi
On 23 Jul 2008, at 19:47, Jos? Mar?a Fern?ndez Gonz?lez wrote:
> IMHO it can be dangerous for the MOBY payload integrity using
> XInclude at mobyContent childs (mobyData) level, but it can be
> needed in scenarios like the one described by Pieter. For security
> reasons, XInclude / XLink / XPointer should be restricted to the
> contents of mobyData, not to MOBY Data itself. For instance,
> references to Simples or Collections, Collections composed by a
> mixure of Simples and references to Simples, or Secondary parameters
> copied from run to run.
>
> Best Regards,
> Jos? Mar?a
>
> Mark Wilkinson wrote:
>> On Wed, 23 Jul 2008 08:06:38 -0700, Pieter Neerincx > > wrote:
>>> That doesn't sound very appealing to me. I use quite large data
>>> structures and can have tens of thousands of them in a collection
>>> (think of oligos for micro arrays.) If I can only replace the
>>> primitives with references, I would still have to send a BioMoby
>>> XML structure with hundreds of thousands of references replacing
>>> the primitives.
>> Hmmm... that's a situation that we had NEVER considered! Putting
>> xlinks into a component of the Moby message higher than the
>> mobyData block was not a scenario that came up during our
>> discussions in Tokyo. the xlink was really only intended to carry
>> data, not data + moby message structure... but I suppose it COULD!
>> How do others feel about this?
>> M
>
> --
> "There is no reason why anybody would want a computer in their home" -
> Ken Olson, founder of DEC 1977
> "640K ought to be enough for anybody" - Bill Gates, 1981
> "Nobody will ever outgrow a 20Mb hard drive." - ???
>
> "Premature optimization is the root of all evil." - Donald Knuth
>
> Jos? Mar?a Fern?ndez Gonz?lez
> Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
> e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
> Unidad del Instituto Nacional de Bioinform?tica
> Biolog?a Estructural y Biocomputaci?n Structural Biology and
> Biocomputing
> Centro Nacional de Investigaciones Oncol?gicas
> C.P.: 28029 Zip Code: 28029
> C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
>
> **NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso
> los ficheros adjuntos, pueden contener informaci?n protegida para el
> uso exclusivo de su destinatario. Se proh?be la distribuci?n,
> reproducci?n o cualquier otro tipo de transmisi?n por parte de otra
> persona que no sea el destinatario. Si usted recibe por error este
> correo, se ruega comunicarlo al remitente y borrar el mensaje
> recibido.
> **CONFIDENTIALITY NOTICE** This email communication and any
> attachments may contain confidential and privileged information for
> the sole use of the designated recipient named above. Distribution,
> reproduction or any other use of this transmission by any party
> other than the intended recipient is prohibited. If you are not the
> intended recipient please contact the sender and delete all copies.
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
-------------------------------------------------------------
From martin.senger at gmail.com Thu Jul 24 00:41:35 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Thu, 24 Jul 2008 08:41:35 +0800
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <48872C42.3060008@cnio.es>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
<488729f9.20f8720a.0551.0626@mx.google.com> <48872C42.3060008@cnio.es>
Message-ID: <4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com>
> I have realized that there are two sets of jMoby artifacts.
I can't see them. Where should I look?
The official jMoby Maven representation is in
http://biomoby.org/m2repo/org/biomoby/jmoby/ and its headers are:
org.biomoby
jmoby
1.1.0
There is also a Taverna-related artifact, here:
http://biomoby.org/m2repo/org/biomoby/taverna-for-moby/. But it may be too
old and obsolete (version 2004). This one has, indeed, a different group ID
(biomoby.org) which is a bit unfortunate (because it is confusing).
Anything I should change?
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From edward.kawas at gmail.com Thu Jul 24 00:47:45 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 23 Jul 2008 17:47:45 -0700
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com> <488729f9.20f8720a.0551.0626@mx.google.com>
<48872C42.3060008@cnio.es>
<4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com>
Message-ID: <4887d13d.02578c0a.4963.ffffe502@mx.google.com>
Hey Martin,
The one that Jose Maria is referring to is the one on
http://moby.ucalgary.ca/moby_maven/biomoby/org/
That was the one that I was using for the taverna-biomoby plugin.
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
Sent: July-23-08 5:42 PM
To: Core developer announcements
Subject: Re: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
> I have realized that there are two sets of jMoby artifacts.
I can't see them. Where should I look?
The official jMoby Maven representation is in
http://biomoby.org/m2repo/org/biomoby/jmoby/ and its headers are:
org.biomoby
jmoby
1.1.0
There is also a Taverna-related artifact, here:
http://biomoby.org/m2repo/org/biomoby/taverna-for-moby/. But it may be too
old and obsolete (version 2004). This one has, indeed, a different group ID
(biomoby.org) which is a bit unfortunate (because it is confusing).
Anything I should change?
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From martin.senger at gmail.com Thu Jul 24 00:58:02 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Thu, 24 Jul 2008 08:58:02 +0800
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <4887d13d.02578c0a.4963.ffffe502@mx.google.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
<488729f9.20f8720a.0551.0626@mx.google.com> <48872C42.3060008@cnio.es>
<4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com>
<4887d13d.02578c0a.4963.ffffe502@mx.google.com>
Message-ID: <4d93f07c0807231758n6f82f337o2cf7d5ced2d60a76@mail.gmail.com>
>
> The one that Jose Maria is referring to is the one on
> http://moby.ucalgary.ca/moby_maven/biomoby/org/
>
> That was the one that I was using for the taverna-biomoby plugin.
Okay, splendid. Why always must we two, Paul and I, have things
differently/separately? Or perhaps it was not Paul who created this
repository...
We should use one Maven repository that was created for us at open-bio
servers. At least, that is where I am putting jMoby releases. The
taverna-plugin should be there, as well (my opinion).
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From edward.kawas at gmail.com Thu Jul 24 01:01:34 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 23 Jul 2008 18:01:34 -0700
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <4d93f07c0807231758n6f82f337o2cf7d5ced2d60a76@mail.gmail.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com> <488729f9.20f8720a.0551.0626@mx.google.com>
<48872C42.3060008@cnio.es> <4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com> <4887d13d.02578c0a.4963.ffffe502@mx.google.com>
<4d93f07c0807231758n6f82f337o2cf7d5ced2d60a76@mail.gmail.com>
Message-ID: <4887d47a.08b38c0a.380d.fffff846@mx.google.com>
Hey Martin,
Shoot me ;-)
That repository was created by me when taverna moved over to supporting
maven as a way for me to update the core jmoby code that the plugin uses.
Like I said previously, I want to kill that ucalgary repository once the
current one has the code that I need.
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
Sent: July-23-08 5:58 PM
To: Core developer announcements
Subject: Re: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
>
> The one that Jose Maria is referring to is the one on
> http://moby.ucalgary.ca/moby_maven/biomoby/org/
>
> That was the one that I was using for the taverna-biomoby plugin.
Okay, splendid. Why always must we two, Paul and I, have things
differently/separately? Or perhaps it was not Paul who created this
repository...
We should use one Maven repository that was created for us at open-bio
servers. At least, that is where I am putting jMoby releases. The
taverna-plugin should be there, as well (my opinion).
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From markw at illuminae.com Thu Jul 24 01:04:31 2008
From: markw at illuminae.com (markw at illuminae.com)
Date: Thu, 24 Jul 2008 01:04:31 +0000
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <4887d47a.08b38c0a.380d.fffff846@mx.google.com>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com> <488729f9.20f8720a.0551.0626@mx.google.com><48872C42.3060008@cnio.es> <4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com> <4887d13d.02578c0a.4963.ffffe502@mx.google.com><4d93f07c0807231758n6f82f337o2cf7d5ced2d60a76@mail.gmail.com><4887d47a.08b38c0a.380d.fffff846@mx.google.com>
Message-ID: <2124445639-1216861474-cardhu_decombobulator_blackberry.rim.net-1399312485-@bxe175.bisx.prod.on.blackberry>
Eddie, be careful! Martin might just take you up on the offer! He isn't always as sweet and gentle as he is when he's in the beautiful city of Vancouver! LOL!
We love you Martin :-)
M
On the Road!
-----Original Message-----
From: "Edward Kawas"
Date: Wed, 23 Jul 2008 18:01:34
To: 'Core developer announcements'
Subject: Re: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
Hey Martin,
Shoot me ;-)
That repository was created by me when taverna moved over to supporting
maven as a way for me to update the core jmoby code that the plugin uses.
Like I said previously, I want to kill that ucalgary repository once the
current one has the code that I need.
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
Sent: July-23-08 5:58 PM
To: Core developer announcements
Subject: Re: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
>
> The one that Jose Maria is referring to is the one on
> http://moby.ucalgary.ca/moby_maven/biomoby/org/
>
> That was the one that I was using for the taverna-biomoby plugin.
Okay, splendid. Why always must we two, Paul and I, have things
differently/separately? Or perhaps it was not Paul who created this
repository...
We should use one Maven repository that was created for us at open-bio
servers. At least, that is where I am putting jMoby releases. The
taverna-plugin should be there, as well (my opinion).
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From martin.senger at gmail.com Thu Jul 24 01:35:59 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Thu, 24 Jul 2008 09:35:59 +0800
Subject: [MOBY-dev] jMoby - version 1.1.0 deployed to Maven
In-Reply-To: <2124445639-1216861474-cardhu_decombobulator_blackberry.rim.net-1399312485-@bxe175.bisx.prod.on.blackberry>
References: <4d93f07c0807230235t2f10f43aq9776c1f613680436@mail.gmail.com>
<488729f9.20f8720a.0551.0626@mx.google.com> <48872C42.3060008@cnio.es>
<4d93f07c0807231741l60d97d6dyf0ccf7d5d53f5462@mail.gmail.com>
<4887d13d.02578c0a.4963.ffffe502@mx.google.com>
<4d93f07c0807231758n6f82f337o2cf7d5ced2d60a76@mail.gmail.com>
<4887d47a.08b38c0a.380d.fffff846@mx.google.com>
<2124445639-1216861474-cardhu_decombobulator_blackberry.rim.net-1399312485-@bxe175.bisx.prod.on.blackberry>
Message-ID: <4d93f07c0807231835y3b9457d2q65728de32d7b6c17@mail.gmail.com>
:-) - I promise: no shooting!
Thanks for all clarification (and, BTW, thanks for wonderful days in
Vancouver, I really, really, enjoyed it).
Cheers,
Martin
PS. Eddie, let me know if you need any assistance with the [re]packaging of
the jMoby libraries. M.
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From jmfernandez at cnio.es Thu Jul 24 13:34:05 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Thu, 24 Jul 2008 15:34:05 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
Message-ID: <488884CD.9030403@cnio.es>
Hi Pieter,
I agree with you about the references to primary Simple and Collection
articles: they are absolutely needed. References to Secondary parameters can
be needed to copy/use the same values in other calls. I am not strongly
against references to full mobyData elements, but they should be considered
and used very carefully, because they can add more mobyData elements than you
wanted, and you cannot check duplicity of queryIds until all the referenced
mobyData elements are fetched.
With an ill-designed service (or libraries) and a carefully built XInclude
reference, with no checks local contents (from the computer or the intranet)
could be referenced and perhaps obtained. With an even more ill-designed
service (or libraries) and a carefully built XInclude reference, an external
hacker could fire OOM (Out Of Memory) or buffer overflow conditions.
Best Regards,
Jos? Mar?a
Pieter Neerincx wrote:
> Hi Jos?,
>
> I don't understand how references could be security risk. And if they do
> why the level at which you could use them would impact security. Could
> you please explain?
>
> If references to mobyData blocks are not an option having at least
> references to BioMoby articles would already be a big help. Secondaries
> are usually very small so I wouldn't need / use references there, but
> for primary Simple and especially for primary Collection articles it
> would be a big plus!
>
> Cheers,
>
> Pi
>
> On 23 Jul 2008, at 19:47, Jos? Mar?a Fern?ndez Gonz?lez wrote:
>
>> IMHO it can be dangerous for the MOBY payload integrity using XInclude
>> at mobyContent childs (mobyData) level, but it can be needed in
>> scenarios like the one described by Pieter. For security reasons,
>> XInclude / XLink / XPointer should be restricted to the contents of
>> mobyData, not to MOBY Data itself. For instance, references to Simples
>> or Collections, Collections composed by a mixure of Simples and
>> references to Simples, or Secondary parameters copied from run to run.
>>
>> Best Regards,
>> Jos? Mar?a
>>
>> Mark Wilkinson wrote:
>>> On Wed, 23 Jul 2008 08:06:38 -0700, Pieter Neerincx
>>> wrote:
>>>> That doesn't sound very appealing to me. I use quite large data
>>>> structures and can have tens of thousands of them in a collection
>>>> (think of oligos for micro arrays.) If I can only replace the
>>>> primitives with references, I would still have to send a BioMoby XML
>>>> structure with hundreds of thousands of references replacing the
>>>> primitives.
>>> Hmmm... that's a situation that we had NEVER considered! Putting
>>> xlinks into a component of the Moby message higher than the mobyData
>>> block was not a scenario that came up during our discussions in
>>> Tokyo. the xlink was really only intended to carry data, not data +
>>> moby message structure... but I suppose it COULD!
>>> How do others feel about this?
>>> M
>>
>> --
>> "There is no reason why anybody would want a computer in their home" -
>> Ken Olson, founder of DEC 1977
>> "640K ought to be enough for anybody" - Bill Gates, 1981
>> "Nobody will ever outgrow a 20Mb hard drive." - ???
>>
>> "Premature optimization is the root of all evil." - Donald Knuth
>>
>> Jos? Mar?a Fern?ndez Gonz?lez
>> Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
>> e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
>> Unidad del Instituto Nacional de Bioinform?tica
>> Biolog?a Estructural y Biocomputaci?n Structural Biology and
>> Biocomputing
>> Centro Nacional de Investigaciones Oncol?gicas
>> C.P.: 28029 Zip Code: 28029
>> C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
>>
>> **NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los
>> ficheros adjuntos, pueden contener informaci?n protegida para el uso
>> exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n
>> o cualquier otro tipo de transmisi?n por parte de otra persona que no
>> sea el destinatario. Si usted recibe por error este correo, se ruega
>> comunicarlo al remitente y borrar el mensaje recibido.
>> **CONFIDENTIALITY NOTICE** This email communication and any
>> attachments may contain confidential and privileged information for
>> the sole use of the designated recipient named above. Distribution,
>> reproduction or any other use of this transmission by any party other
>> than the intended recipient is prohibited. If you are not the intended
>> recipient please contact the sender and delete all copies.
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> -------------------------------------------------------------
> Wageningen University and Research centre (WUR)
> Laboratory of Bioinformatics
> Transitorium (building 312) room 1034
>
> Dreijenlaan 3
> 6703 HA Wageningen
> The Netherlands
>
> phone: +31 (0)317-483 060
> mobile: +31 (0)6-143 66 783
> e-mail: pieter.neerincx at gmail.com
> skype: pieter.online
> -------------------------------------------------------------
>
>
>
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.
From martin.senger at gmail.com Fri Jul 25 00:06:03 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Fri, 25 Jul 2008 08:06:03 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <488884CD.9030403@cnio.es>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
Message-ID: <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
Thank you for all your comments. We need to move on, however. I am, almost
impatiently, waiting for a conclusion because I need to implement the
"data-by-reference" rather soon. Are there other people preparing/building
their comments - so we should wait for them, or can I summarize what was
said so far - and create an official "RFC" from it?
My personal opinion is still on the side to have references only for
primitives. I am not convinced by the Peter's vision of a huge collection
that is too big even when it uses references. I am simply trying to find a
solution that will be good for the most cases - and then deal with the
marginal cases separately (like by designing the service differently).
I would like to hear also Eddie's voice - because he knows how easy or hard
it would be to make the way we decide to do the "data-by-reference" in
Taverna (and I am still talking about the T1 which I expect to be supported
for some time).
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Fri Jul 25 09:25:31 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Fri, 25 Jul 2008 11:25:31 +0200
Subject: [MOBY-dev] findService and the hierarchy
In-Reply-To: <4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
References: <487C9061.3030203@mpiz-koeln.mpg.de>
<4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
Message-ID: <48899C0B.70408@mpiz-koeln.mpg.de>
i like to bring this up for discussion again ;-)
thanks
Martin Senger wrote:
>> In my opinion when searching for services consuming Annotated_GO_Term via
>> the findService method i should also include the service which consumes a
>> GO_Term datatype, as this is the parent datatype (per default
>> 'includeParentDatatype' is true)....
>>
>>
>
> I assume that by "I should also include" you actually meant "the registry
> response should also include". Is my understanding correct?
>
> If so, I am afraid that it is not the issue of the jMoby library but the
> registry itself. Eddie or Mark can confirm or reject this hypothesis.
>
> The BioMoby API allows to specify two XML tags when calling findService
> method: the
>
> X
> and
> X
>
> Where X can be either 1 or 0.
>
> The jMoby's findService() method sets these two tags according to the two
> boolean parameters that the method can get:
>
> boolean includeChildrenServiceTypes
> and
> boolean includeParentDataTypes
>
> I have tried to call the findService, with the Annotated_GO_Term input,
> using all combinations of these two tags (you can do it using the Moby
> command-line client with the optioins -nc and/or -np). The results are
> either no service or 713 services:
>
> expandObjects expandServices # of found services
> 1 1 713
> 0 0 0
> 1 0 713
> 0 0 0
>
> I have feeling that this is not a correct result. Could Eddie or Mark
> perhaps comment on it?
>
> [I vaguely remember that we had this conversation about these tags with Mark
> years(!) ago - but I do not recall what was the conclusion. I only, and
> again vaguely, remember to have this issue in my to-do list - which would
> indicate that something could be done in the jMoby library.]
>
> Cheers,
> Martin
>
>
--
=====================================================
Dipl. Bioinf. Andreas Groscurth
Bioinformatics Software Developer
Plant Computational Biology group
Max-Planck Institute for plant breeding research
Carl-von-Linne Weg 10
50829 Cologne
Germany
+49(0) 221 5062449
=====================================================
From martin.senger at gmail.com Fri Jul 25 09:34:15 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Fri, 25 Jul 2008 17:34:15 +0800
Subject: [MOBY-dev] findService and the hierarchy
In-Reply-To: <48899C0B.70408@mpiz-koeln.mpg.de>
References: <487C9061.3030203@mpiz-koeln.mpg.de>
<4d93f07c0807222009x7c108d25j52930dae4f9ddaf1@mail.gmail.com>
<48899C0B.70408@mpiz-koeln.mpg.de>
Message-ID: <4d93f07c0807250234h53e049bv9b97edecbd0865e3@mail.gmail.com>
> i like to bring this up for discussion again ;-)
Well, let's not discuss it but let's look what we are getting back from the
registry when different values are given for expandObjects and
expandService. I posted my results - now it is up to Eddie or Mark to draw
conclusion from that results... :-)
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From pieter.neerincx at gmail.com Fri Jul 25 10:50:12 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Fri, 25 Jul 2008 12:50:12 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
Message-ID: <2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
Hi Martin at al.,
On 25?Jul?2008, at 2:06 AM, Martin Senger wrote:
> Thank you for all your comments. We need to move on, however. I am,
> almost
> impatiently, waiting for a conclusion because I need to implement the
> "data-by-reference" rather soon. Are there other people preparing/
> building
> their comments - so we should wait for them, or can I summarize what
> was
> said so far - and create an official "RFC" from it?
>
> My personal opinion is still on the side to have references only for
> primitives. I am not convinced by the Pieter's vision of a huge
> collection
> that is too big even when it uses references. I am simply trying to
> find a
> solution that will be good for the most cases - and then deal with the
> marginal cases separately (like by designing the service differently).
Well, especially if you want to have a solution that works in most
cases, I think we should have pass-by-reference at *any* level in
mobyData. How often will the data of a single primitive be too large
for inclusion in the SOAP body? I can imagine wrapping legacy data
formats like a complete EMBL/Genbank/DDBJ record in a String object or
sending images as base 64 encoded pieces of text in a String object.
Maybe even sending a bulky sequence like a complete chromosome inside
a String object, but in most cases the raw data carried by a primitive
will be smaller than your average URL! And even if you do send some
bulky stuff inside a String object as long as you send only one of
those at a time it'll still work in most cases. But once you start to
send tens, hundreds or more of those you have a problem. If you still
send around tens, hundreds or more URLs, you'd have to fetch each one
individually causing massive overhead. The reason to implement pass-by-
reference is to make BioMoby scale much better for big jobs and in
such cases fetching data from a single reference is much more efficient.
Take for example position information of alignments on a genome:
13
97419218
97419282
65M
With pass-by-reference only for primitives this would become something
like this:
That ain't much of an improvement!
I have on average 30.000 features on a micro array and for each oligo
on average 3 hits on a reference assembly. That makes for an average
total of 90.000 HitPosition objects and that is just a small part of
the annotation for my oligos....
Just to stress that these are not hypothetical cases: The stuff above
is just a single example of what I'm already using for more than two
years. Off course I needed a pass by reference workaround, because the
data is too big for the SOAP body. So I registered an URL object and
send those around. These URL objects point to chunks of BioMoby XML
which is a complete mobyData block in most cases. Although the URL
obeject is fully compatible with the current BioMoby standard, it's an
ugly solution because of two reasons:
1. It's not a standard to do pass-by-reference.
2. It defeats the entire purpose of having the BioMoby object ontology
to improve automatic service discovery. You can only discover that I
provide several services which consume or produce URL objects, but you
can not discover automatically what those URLs point to, so most of
the URL object producers and consumers will be incompatible! (Ok, I
can use namespace restriction to limit the problem of incompatible
services a bit, but still you would have no idea what the URL points
to based on the data in BioMoby Central.)
So I would love to see standardised pass-by-reference as part of the
BioMoby specs and I think it doesn't require rocket science to do this
at any level in the structure of BioMoby objects. Why don't we simply
do the following:
Current situation:
A BioMoby object is a BioMoby tripple with optional articleName
attribute and optionally raw character data for primitives. The
tripple is the XML element name, an id attribute and a namespace
attribute.
New situation:
The above + The id and namespace attributes of a BioMoby tripple can
be replaced with an xlink attribute resulting in a "BioMoby double".
If the latter is the case the element containing the xlink attribute
and all it's children are available from link specified by the xlink
attribute.
Example old:
ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
4569343
Example with pass-by-reference:
which points to:
ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
4569343
Hence note that the link points to a ComplexObject and not just to
it's children. The latter would also be an option but than you would
get:
which points to:
ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
4569343
If you have more than one child element like with the example above,
some XML parsers might have problems with such a chunk. Although it is
well balanced, it doesn't have a (pseudo-)root element. So for
practical reasons I suggest to have the links point to the element
containing the xlink attribute and it's children. This should be
really easy to parse. You either have an id + namespace attribute or
you have an xlink attribute. If both were not present the values were
NULL. This doesn't require parsing dozens of WSRF tags in a header. It
even doesn't require a service to tell the client using the
serviceNotes or something similar that it did pass-by-reference of
some kind nor does it require the client to specify that it can
understand certain references. Off course it would be handy though to
have some extension to BioMoby Central to prevent discovering services
providing references which are incompatible with your client.
Secondly, if a service can provide multiple types of references and
your client doesn't understand them all, it would also be nice if a
client can specify a preference for a certain type of reference. But
both wouldn't be required for a first quick implementation of pass-by-
reference.
> I would like to hear also Eddie's voice - because he knows how easy
> or hard
> it would be to make the way we decide to do the "data-by-reference" in
> Taverna (and I am still talking about the T1 which I expect to be
> supported
> for some time).
I agree we need Eddie's feedback on Taverna compatibility!
Compatibility with Taverna 1 would be great, but it would be extremely
lame if our BioMoby references turn out to be incompatible with the
new pass-by-reference feature of Taverna 2.
I hope you see the potential for improving scalability of BioMoby
services with pass-by-reference at any level of mobyData!
Cheers,
Pi
> Martin
>
> --
> Martin Senger
> email: martin.senger at gmail.com,m.senger at cgiar.org
> skype: martinsenger
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
------------------------------------------------------------
From martin.senger at gmail.com Fri Jul 25 11:14:01 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Fri, 25 Jul 2008 19:14:01 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
Message-ID: <4d93f07c0807250414q1d456695ld65cd3d5a6daa63@mail.gmail.com>
Pieter, thanks! Your reply was very good, and convincing! I like the most
the fact that you are driven by a real use case, and a long one, lasting
already for two years.
I had in mind other use cases, but now I agree that Pieter's point of view
is better. I will read his email once again, just to be sure that I have not
missed something.
Are we closer now to the RFC proposal?
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Fri Jul 25 11:15:52 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Fri, 25 Jul 2008 13:15:52 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es> <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com> <488884CD.9030403@cnio.es> <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
Message-ID: <4889B5E8.6010903@mpiz-koeln.mpg.de>
Hi,
I'm not sure whether i get the complete point of the discussion here,
nevertheless I like to jump in into that.
What i dont understand is why do we need references for only primitives
? I dont understand what a reference for a primitive is good for ?
I would also like to have references for any type, if possible.
Currently i work on sending image information around. some images are up
to several MBs. It would be way more convinient to send references to
the http url of the image instead of encoding/decoding it and
transferring around some networks.
If any type of object could be a reference, then we could stay with the
current way of registering and discovering services....
It might also just the case i dont get the point what martin said ;-)
Cheers
andreas
Pieter Neerincx wrote:
> Hi Martin at al.,
>
> On 25?Jul?2008, at 2:06 AM, Martin Senger wrote:
>
>> Thank you for all your comments. We need to move on, however. I am,
>> almost
>> impatiently, waiting for a conclusion because I need to implement the
>> "data-by-reference" rather soon. Are there other people
>> preparing/building
>> their comments - so we should wait for them, or can I summarize what was
>> said so far - and create an official "RFC" from it?
>>
>> My personal opinion is still on the side to have references only for
>> primitives. I am not convinced by the Pieter's vision of a huge
>> collection
>> that is too big even when it uses references. I am simply trying to
>> find a
>> solution that will be good for the most cases - and then deal with the
>> marginal cases separately (like by designing the service differently).
>
> Well, especially if you want to have a solution that works in most
> cases, I think we should have pass-by-reference at *any* level in
> mobyData. How often will the data of a single primitive be too large
> for inclusion in the SOAP body? I can imagine wrapping legacy data
> formats like a complete EMBL/Genbank/DDBJ record in a String object or
> sending images as base 64 encoded pieces of text in a String object.
> Maybe even sending a bulky sequence like a complete chromosome inside
> a String object, but in most cases the raw data carried by a primitive
> will be smaller than your average URL! And even if you do send some
> bulky stuff inside a String object as long as you send only one of
> those at a time it'll still work in most cases. But once you start to
> send tens, hundreds or more of those you have a problem. If you still
> send around tens, hundreds or more URLs, you'd have to fetch each one
> individually causing massive overhead. The reason to implement
> pass-by-reference is to make BioMoby scale much better for big jobs
> and in such cases fetching data from a single reference is much more
> efficient.
>
> Take for example position information of alignments on a genome:
>
> moby:namespace='transcriptome' moby:articleName='hit_position'>
> moby:articleName='seq_id'> 13
> moby:articleName='start'> 97419218
> moby:articleName='stop'> 97419282
> moby:articleName='strand'/>
> moby:articleName='cll'> 65M
>
>
> With pass-by-reference only for primitives this would become something
> like this:
>
> moby:namespace='transcriptome' moby:articleName='hit_position'>
> moby:articleName='seq_id'
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/String_seq_id.xml'/>
>
> moby:articleName='start'
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/Integer_start.xml'/>
>
> moby:articleName='stop'
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/Integer_stop.xml'/>
> moby:articleName='strand'/>
> moby:articleName='cll'
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/String_cll.xml'/>
>
>
> That ain't much of an improvement!
>
> I have on average 30.000 features on a micro array and for each oligo
> on average 3 hits on a reference assembly. That makes for an average
> total of 90.000 HitPosition objects and that is just a small part of
> the annotation for my oligos....
>
> Just to stress that these are not hypothetical cases: The stuff above
> is just a single example of what I'm already using for more than two
> years. Off course I needed a pass by reference workaround, because the
> data is too big for the SOAP body. So I registered an URL object and
> send those around. These URL objects point to chunks of BioMoby XML
> which is a complete mobyData block in most cases. Although the URL
> obeject is fully compatible with the current BioMoby standard, it's an
> ugly solution because of two reasons:
> 1. It's not a standard to do pass-by-reference.
> 2. It defeats the entire purpose of having the BioMoby object ontology
> to improve automatic service discovery. You can only discover that I
> provide several services which consume or produce URL objects, but you
> can not discover automatically what those URLs point to, so most of
> the URL object producers and consumers will be incompatible! (Ok, I
> can use namespace restriction to limit the problem of incompatible
> services a bit, but still you would have no idea what the URL points
> to based on the data in BioMoby Central.)
>
> So I would love to see standardised pass-by-reference as part of the
> BioMoby specs and I think it doesn't require rocket science to do this
> at any level in the structure of BioMoby objects. Why don't we simply
> do the following:
>
> Current situation:
>
> A BioMoby object is a BioMoby tripple with optional articleName
> attribute and optionally raw character data for primitives. The
> tripple is the XML element name, an id attribute and a namespace
> attribute.
>
> New situation:
> The above + The id and namespace attributes of a BioMoby tripple can
> be replaced with an xlink attribute resulting in a "BioMoby double".
> If the latter is the case the element containing the xlink attribute
> and all it's children are available from link specified by the xlink
> attribute.
>
> Example old:
>
> articleName='MyFavoriteObject'>
>
> ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
>
>
> 4569343
>
>
>
> Example with pass-by-reference:
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/ComplexObject.xml' articleName='MyFavoriteObject'
> />
>
> which points to:
>
> articleName='MyFavoriteObject'>
>
> ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
>
>
> 4569343
>
>
>
> Hence note that the link points to a ComplexObject and not just to
> it's children. The latter would also be an option but than you would get:
>
> xlink='http://www.mydomain.org/biomoby/tmp/job1239573/ComplexObject_content.xml' id='accession_number123875'
> namespace='' articleName='MyFavoriteObject' />
>
> which points to:
>
>
> ATTGCGCGCTAGAGTGCGGGTGTGCAAACCGGTGT
>
>
> 4569343
>
>
> If you have more than one child element like with the example above,
> some XML parsers might have problems with such a chunk. Although it is
> well balanced, it doesn't have a (pseudo-)root element. So for
> practical reasons I suggest to have the links point to the element
> containing the xlink attribute and it's children. This should be
> really easy to parse. You either have an id + namespace attribute or
> you have an xlink attribute. If both were not present the values were
> NULL. This doesn't require parsing dozens of WSRF tags in a header. It
> even doesn't require a service to tell the client using the
> serviceNotes or something similar that it did pass-by-reference of
> some kind nor does it require the client to specify that it can
> understand certain references. Off course it would be handy though to
> have some extension to BioMoby Central to prevent discovering services
> providing references which are incompatible with your client.
> Secondly, if a service can provide multiple types of references and
> your client doesn't understand them all, it would also be nice if a
> client can specify a preference for a certain type of reference. But
> both wouldn't be required for a first quick implementation of
> pass-by-reference.
>
>> I would like to hear also Eddie's voice - because he knows how easy
>> or hard
>> it would be to make the way we decide to do the "data-by-reference" in
>> Taverna (and I am still talking about the T1 which I expect to be
>> supported
>> for some time).
>
> I agree we need Eddie's feedback on Taverna compatibility!
> Compatibility with Taverna 1 would be great, but it would be extremely
> lame if our BioMoby references turn out to be incompatible with the
> new pass-by-reference feature of Taverna 2.
>
> I hope you see the potential for improving scalability of BioMoby
> services with pass-by-reference at any level of mobyData!
>
> Cheers,
>
> Pi
>
>> Martin
>>
>> --
>> Martin Senger
>> email: martin.senger at gmail.com,m.senger at cgiar.org
>> skype: martinsenger
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> -------------------------------------------------------------
> Wageningen University and Research centre (WUR)
> Laboratory of Bioinformatics
> Transitorium (building 312) room 1034
>
> Dreijenlaan 3
> 6703 HA Wageningen
> The Netherlands
>
> phone: +31 (0)317-483 060
> mobile: +31 (0)6-143 66 783
> e-mail: pieter.neerincx at gmail.com
> skype: pieter.online
> ------------------------------------------------------------
>
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
--
=====================================================
Dipl. Bioinf. Andreas Groscurth
Bioinformatics Software Developer
Plant Computational Biology group
Max-Planck Institute for plant breeding research
Carl-von-Linne Weg 10
50829 Cologne
Germany
+49(0) 221 5062449
=====================================================
From martin.senger at gmail.com Fri Jul 25 11:35:33 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Fri, 25 Jul 2008 19:35:33 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4889B5E8.6010903@mpiz-koeln.mpg.de>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
<4889B5E8.6010903@mpiz-koeln.mpg.de>
Message-ID: <4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
> What i dont understand is why do we need references for only primitives ? I
> dont understand what a reference for a primitive is good for ?
If you do not understand, it is my fault - I have tried to explain it, but
obviously have not done it well.
The discussion is now, however, a bit farer.Your first question is probably
obsolete - the forming RFC seems to be that the references will be able to
be anywhere (or not only on primitives).
Your second question: We need to allow reference on primitives if we want to
reuse existing references on not-yet-moby objects (such as uniprot
sequences).
The same with URLs of existing images - as in your use case. What you are
describing is exactly what we are going to do. You will have a choice: a)
You store somewhere an XML moby message (its part) and send a reference to
it, or b) you store somewhere only the image (not wrapped in a moby message)
and you send a moby message with a reference attached to a primitive.
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From groscurt at mpiz-koeln.mpg.de Fri Jul 25 11:46:45 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Fri, 25 Jul 2008 13:46:45 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es> <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com> <488884CD.9030403@cnio.es> <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com> <2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com> <4889B5E8.6010903@mpiz-koeln.mpg.de>
<4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
Message-ID: <4889BD25.3080701@mpiz-koeln.mpg.de>
Martin Senger wrote:
> If you do not understand, it is my fault - I have tried to explain it, but
> obviously have not done it well.
>
hehe not automatically - i might just missed something ;-)
Thanks for the explanation, now i know i did understand it correctly ;-)
Andreas
> The discussion is now, however, a bit farer.Your first question is probably
> obsolete - the forming RFC seems to be that the references will be able to
> be anywhere (or not only on primitives).
>
> Your second question: We need to allow reference on primitives if we want to
> reuse existing references on not-yet-moby objects (such as uniprot
> sequences).
>
> The same with URLs of existing images - as in your use case. What you are
> describing is exactly what we are going to do. You will have a choice: a)
> You store somewhere an XML moby message (its part) and send a reference to
> it, or b) you store somewhere only the image (not wrapped in a moby message)
> and you send a moby message with a reference attached to a primitive.
>
> Martin
>
>
--
=====================================================
Dipl. Bioinf. Andreas Groscurth
Bioinformatics Software Developer
Plant Computational Biology group
Max-Planck Institute for plant breeding research
Carl-von-Linne Weg 10
50829 Cologne
Germany
+49(0) 221 5062449
=====================================================
From pieter.neerincx at gmail.com Fri Jul 25 12:28:43 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Fri, 25 Jul 2008 14:28:43 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
<4889B5E8.6010903@mpiz-koeln.mpg.de>
<4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
Message-ID:
Hi,
On 25?Jul?2008, at 1:35 PM, Martin Senger wrote:
>> What i dont understand is why do we need references for only
>> primitives ? I
>> dont understand what a reference for a primitive is good for ?
>
>
> If you do not understand, it is my fault - I have tried to explain
> it, but
> obviously have not done it well.
>
> The discussion is now, however, a bit farer.Your first question is
> probably
> obsolete - the forming RFC seems to be that the references will be
> able to
> be anywhere (or not only on primitives).
>
> Your second question: We need to allow reference on primitives if we
> want to
> reuse existing references on not-yet-moby objects (such as uniprot
> sequences).
Yes :) !
> The same with URLs of existing images - as in your use case. What
> you are
> describing is exactly what we are going to do. You will have a
> choice: a)
> You store somewhere an XML moby message (its part) and send a
> reference to
> it, or b) you store somewhere only the image (not wrapped in a moby
> message)
> and you send a moby message with a reference attached to a primitive.
Eh, but in the case of images they must be encoded to put then inside
a String primitive. So, since the data the reference points to must be
the same as in the case where you would not use a reference, it musty
also be encoded. Hence, you can not replace the object with a
reference to a plain (un-encoded) JPEG for example. Take for example
the b64_encoded_jpeg Object. That would look like:
afr78e349tu9rghjerewptu59hgfirhegwug9uwguwujw9-
If you want to replace this with a reference the actual data would
still have to be a base64 encoded piece of text. So you can not make a
reference to an image that exists as a plain JPEG, GIF, etc. on a web
server. We don't have a datatype to send un-encoded binary stuff. You
could create a new (un-encoded) JPEG object and use references, but
than you are required to always send a reference as you can not send
that BioMoby datatype un-encoded as part of the SOAP payload! This is
again not optimal though as you would have different objects for
encoded and un-encoded images limiting the discovery of potentially
compatible services. I couldn't care less about whether it's encoded
or not. I just want to send or receive an image and the optional
encoding is just the side effect of a workaround. I don't see how we
can easily solve that. Does anyone have suggestion how we should
handle this?...
Cheers,
Pi
> Martin
>
> --
> Martin Senger
> email: martin.senger at gmail.com,m.senger at cgiar.org
> skype: martinsenger
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
------------------------------------------------------------
From edward.kawas at gmail.com Fri Jul 25 13:03:34 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Fri, 25 Jul 2008 06:03:34 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es> <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com> <488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
Message-ID: <4889cf38.1aba720a.1b32.3a36@mx.google.com>
Hi Martin,
Don't worry about Taverna! Whatever we decide will most likely be the same
amount of work. So let's just get it right.
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
Sent: July-24-08 5:06 PM
To: Core developer announcements
Subject: Re: [MOBY-dev] data by reference - a request for comments
Thank you for all your comments. We need to move on, however. I am, almost
impatiently, waiting for a conclusion because I need to implement the
"data-by-reference" rather soon. Are there other people preparing/building
their comments - so we should wait for them, or can I summarize what was
said so far - and create an official "RFC" from it?
My personal opinion is still on the side to have references only for
primitives. I am not convinced by the Peter's vision of a huge collection
that is too big even when it uses references. I am simply trying to find a
solution that will be good for the most cases - and then deal with the
marginal cases separately (like by designing the service differently).
I would like to hear also Eddie's voice - because he knows how easy or hard
it would be to make the way we decide to do the "data-by-reference" in
Taverna (and I am still talking about the T1 which I expect to be supported
for some time).
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From gordonp at ucalgary.ca Fri Jul 25 15:05:42 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Fri, 25 Jul 2008 09:05:42 -0600
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4889cf38.1aba720a.1b32.3a36@mx.google.com>
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es> <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com> <488884CD.9030403@cnio.es> <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<4889cf38.1aba720a.1b32.3a36@mx.google.com>
Message-ID: <4889EBC6.6010500@ucalgary.ca>
I'll finally chime in...I'm in favour of data-by-reference at any level
within a mobyData block. Above that and you get weird issues with
exception maintenance, and any restriction below that appears pretty
artificial to me.
Edward Kawas wrote:
> Hi Martin,
>
> Don't worry about Taverna! Whatever we decide will most likely be the same
> amount of work. So let's just get it right.
>
> Eddie
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Martin Senger
> Sent: July-24-08 5:06 PM
> To: Core developer announcements
> Subject: Re: [MOBY-dev] data by reference - a request for comments
>
> Thank you for all your comments. We need to move on, however. I am, almost
> impatiently, waiting for a conclusion because I need to implement the
> "data-by-reference" rather soon. Are there other people preparing/building
> their comments - so we should wait for them, or can I summarize what was
> said so far - and create an official "RFC" from it?
>
> My personal opinion is still on the side to have references only for
> primitives. I am not convinced by the Peter's vision of a huge collection
> that is too big even when it uses references. I am simply trying to find a
> solution that will be good for the most cases - and then deal with the
> marginal cases separately (like by designing the service differently).
>
> I would like to hear also Eddie's voice - because he knows how easy or hard
> it would be to make the way we decide to do the "data-by-reference" in
> Taverna (and I am still talking about the T1 which I expect to be supported
> for some time).
>
> Martin
>
>
From jason.e.stewart at gmail.com Fri Jul 25 16:07:22 2008
From: jason.e.stewart at gmail.com (Jason Stewart)
Date: Fri, 25 Jul 2008 21:37:22 +0530
Subject: [MOBY-dev] Interesting...
In-Reply-To:
References:
Message-ID: <41c1ade50807250907k61a3a459m3d394bf6d793df5f@mail.gmail.com>
Hey Mark,
On Wed, Jul 2, 2008 at 7:52 PM, Mark wrote:
> http://www.biomedcentral.com/1471-2105/9/295/abstract
>
> I wonder how they got document-literal stuff working in Perl??
Can you get me a pdf of that article? It looks like a subscription is needed.
Cheers, jas.
From edward.kawas at gmail.com Fri Jul 25 18:05:23 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Fri, 25 Jul 2008 11:05:23 -0700
Subject: [MOBY-dev] updated wsrf perl module
Message-ID: <488a1605.06d7720a.021c.ffff9cd4@mx.google.com>
Hi all,
I am trying to create a cpan distribution for the WSRF module created by
Mark McKeown. In the process, I wanted to see if I could get it to work with
windows (you never know, there might be some windows moby service developers
out there somewhere ;-) ) and to see if I could improve the install
procedure.
So, I now have a beta version of the module that I plan on sending to cpan,
but I want to make sure that I didn't break anything for those using this
module for creating async services. Is it possible for those that have
developed async services to try out this new module and to let me know if
anything is missing, broken, etc?
I would really appreciate any feedback.
Thanks,
Eddie
-------------- next part --------------
A non-text attachment was scrubbed...
Name: WSRF-Lite-0.8.2.2.1.tar.gz
Type: application/octet-stream
Size: 454887 bytes
Desc: not available
URL:
From markw at illuminae.com Fri Jul 25 18:34:13 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Fri, 25 Jul 2008 11:34:13 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4889EBC6.6010500@ucalgary.ca>
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<4889cf38.1aba720a.1b32.3a36@mx.google.com>
<4889EBC6.6010500@ucalgary.ca>
Message-ID:
On Fri, 25 Jul 2008 08:05:42 -0700, Paul Gordon
wrote:
> I'll finally chime in...I'm in favour of data-by-reference at any level
> within a mobyData block. Above that and you get weird issues with
> exception maintenance, and any restriction below that appears pretty
> artificial to me.
I agree with Paul. I wouldn't go any higher than the mobyData block, but
allowing it at any lower levels seems sensible to me, particularly given
Pieter's use-case.
M
--
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada
From markw at illuminae.com Fri Jul 25 18:33:07 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Fri, 25 Jul 2008 11:33:07 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com>
<4889B5E8.6010903@mpiz-koeln.mpg.de>
<4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
Message-ID:
On Fri, 25 Jul 2008 05:28:43 -0700, Pieter Neerincx
wrote:
> If you want to replace this with a reference the actual data would still
> have to be a base64 encoded piece of text. So you can not make a
> reference to an image that exists as a plain JPEG, GIF, etc. on a web
> server.
Martin and I discussed this while he was here and the behaviour of this
case will be part of his RFC proposal. It seems to me that it would be
silly to b64 encode something that is being passed by reference... but I
agree that to do anything else is a 'hack'. Nevertheless, there are a lot
of 'hacks' in Moby, and this one is one of the more sensible 'hacks' IMO...
M
--
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada
From jason.e.stewart at gmail.com Sat Jul 26 05:52:11 2008
From: jason.e.stewart at gmail.com (Jason Stewart)
Date: Sat, 26 Jul 2008 11:22:11 +0530
Subject: [MOBY-dev] updated wsrf perl module
In-Reply-To: <488a1605.06d7720a.021c.ffff9cd4@mx.google.com>
References: <488a1605.06d7720a.021c.ffff9cd4@mx.google.com>
Message-ID: <41c1ade50807252252r2cc2482euad274ab511dd3421@mail.gmail.com>
Hey,
Cool Eddie - it's a pain to have to install perl modules by hand -
glad you're making this available on CPAN. I don't have any async
services though...
Cheers, jas.
On Fri, Jul 25, 2008 at 11:35 PM, Edward Kawas wrote:
> Hi all,
>
> I am trying to create a cpan distribution for the WSRF module created by
> Mark McKeown.
From dmitry.repchevski at bsc.es Sat Jul 26 10:49:07 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Sat, 26 Jul 2008 12:49:07 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
Message-ID: <488B0123.3040608@bsc.es>
Hello everybody,
WSRF has a way to reference a part of resource using XPath expressions.
It's easy to include it just by updating current asynchronous specification.
The only thing to do is to implement QueryResourceProperties method.
So we can always get any part of mobyData (for practical reason I think
to limit it to MobySimple/MobyCollection).
This way the reference to any remote resource will be standard WSRF
reference (based on WS-Addressing) with XPath expression to get the
interested part of it.
I'm not very good in XPath, but something like this:
String expression =
"/*[local-name()='MOBY']/*[local-name()='mobyContent']/*[local-name()='mobyData'
and @*[local-name()='queryID']='id_01']/*[local-name()='Simple' and
@*[local-name()='articleName']='sequence']"
will give us a "Simple" with articleName = "sequence";
Best regards,
Dmitry
From martin.senger at gmail.com Sat Jul 26 11:52:30 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Sat, 26 Jul 2008 19:52:30 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <488B0123.3040608@bsc.es>
References:
<488B0123.3040608@bsc.es>
Message-ID: <4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
> WSRF has...
Sorry, if you wish to use WSRF - I am out. Happy hacking...
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From martin.senger at gmail.com Mon Jul 28 01:29:34 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Mon, 28 Jul 2008 09:29:34 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
References:
<488B0123.3040608@bsc.es>
<4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
Message-ID: <4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
I am trying to figure out how I would implement the data by reference in
order to achieve the main purpose - not no have all data in memory. I know
how I would do it for the references being on the primitive types level, but
it is less clear to me how I would do it with the references on other
levels. Perhaps you can help me to explain how you would do it (or even
Pieter can explore how he is already doing it)?
For references on the primitive type level, I would do a usual parsing of a
Moby message and when I find a reference, I would resolve it do a local file
and I would pass to my service class the data as a file reference. It would
be up to the service to read the file contents to the memory or not - that
would depend what it needs to do with the data.
However, on the higher level, such as mobyData level, the reference must be
treated differently. If I do the same (resolving the reference to a local
file), I still need to call a parser again to parse the contents of the
local file. Because I still want to give my service already parsed data (and
not a biomoby XML), and because I do not want to have all in memory, I need
to create local files for each primitive type and do the same as above
(passing my service local file references). I do not see much other choices.
Am I missing a point here?
The above is doable, of course. But you see why I wanted to have references
only on the primitive type level. If we have them on any level (as it seems
to be) I need, in my implementation, actually make another (local)
references for the primitive type, anyway. Of course, it is better than the
remote references because I do not need to make a network connection for
each individual primitive type (point taken, Pieter) - but it still to be
done.
Cheers,
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From gordonp at ucalgary.ca Mon Jul 28 01:50:30 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Sun, 27 Jul 2008 19:50:30 -0600
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
References: <488B0123.3040608@bsc.es> <4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
<4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
Message-ID: <488D25E6.7030601@ucalgary.ca>
Hi Martin,
> I am trying to figure out how I would implement the data by reference in
> order to achieve the main purpose - not no have all data in memory.
I think you are coming at this from a different side than most of us:
data-by-reference is primarily to reduce the bandwidth burden by not
requiring results to go back to the client before being sent to another
service.
Paul
From markw at illuminae.com Mon Jul 28 03:58:56 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Sun, 27 Jul 2008 20:58:56 -0700
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <488D25E6.7030601@ucalgary.ca>
References:
<488B0123.3040608@bsc.es>
<4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
<4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
<488D25E6.7030601@ucalgary.ca>
Message-ID:
On Sun, 27 Jul 2008 18:50:30 -0700, Paul Gordon
wrote:
> Hi Martin,
>> I am trying to figure out how I would implement the data by reference in
>> order to achieve the main purpose - not no have all data in memory.
> I think you are coming at this from a different side than most of us:
> data-by-reference is primarily to reduce the bandwidth burden by not
> requiring results to go back to the client before being sent to another
> service.
I don't think that is the PRIMARY reason, necessarily - we also want to
get around passing large objects in the SOAP message due to limitations in
the (Perl) SOAP libraries, and increasingly in the non-SOAP services where
passing large data inside of an XML tag doesn't allow for nice download
optimizations/caching, etc.
M
From gordonp at ucalgary.ca Mon Jul 28 04:03:21 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Sun, 27 Jul 2008 22:03:21 -0600
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References: <488B0123.3040608@bsc.es> <4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com> <4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com> <488D25E6.7030601@ucalgary.ca>
Message-ID: <488D4509.6070407@ucalgary.ca>
Perhaps we should make a statement of intendion before hammering out the
details then ? ;-)
Mark Wilkinson wrote:
> On Sun, 27 Jul 2008 18:50:30 -0700, Paul Gordon
> wrote:
>
>> Hi Martin,
>>> I am trying to figure out how I would implement the data by
>>> reference in
>>> order to achieve the main purpose - not no have all data in memory.
>> I think you are coming at this from a different side than most of us:
>> data-by-reference is primarily to reduce the bandwidth burden by not
>> requiring results to go back to the client before being sent to
>> another service.
>
> I don't think that is the PRIMARY reason, necessarily - we also want
> to get around passing large objects in the SOAP message due to
> limitations in the (Perl) SOAP libraries, and increasingly in the
> non-SOAP services where passing large data inside of an XML tag
> doesn't allow for nice download optimizations/caching, etc.
>
> M
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> !DSPAM:60005,488d415d1095014969098!
>
>
>
From gordonp at ucalgary.ca Mon Jul 28 04:07:14 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Sun, 27 Jul 2008 22:07:14 -0600
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <488D4509.6070407@ucalgary.ca>
References: <488B0123.3040608@bsc.es> <4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com> <4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com> <488D25E6.7030601@ucalgary.ca>
<488D4509.6070407@ucalgary.ca>
Message-ID: <488D45F2.40900@ucalgary.ca>
Okay, this is a very long thread, so I forgot the original message,
which indeed has both of these uses (except I don't limit my vision of
inter-service data passing to well-known entities as Martin gave in his
example).
Paul Gordon wrote:
> Perhaps we should make a statement of intendion before hammering out
> the details then ? ;-)
>
> Mark Wilkinson wrote:
>> On Sun, 27 Jul 2008 18:50:30 -0700, Paul Gordon
>> wrote:
>>
>>> Hi Martin,
>>>> I am trying to figure out how I would implement the data by
>>>> reference in
>>>> order to achieve the main purpose - not no have all data in memory.
>>> I think you are coming at this from a different side than most of
>>> us: data-by-reference is primarily to reduce the bandwidth burden by
>>> not requiring results to go back to the client before being sent to
>>> another service.
>>
>> I don't think that is the PRIMARY reason, necessarily - we also want
>> to get around passing large objects in the SOAP message due to
>> limitations in the (Perl) SOAP libraries, and increasingly in the
>> non-SOAP services where passing large data inside of an XML tag
>> doesn't allow for nice download optimizations/caching, etc.
>>
>> M
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>>
>>
>>
>>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> !DSPAM:60005,488d4268109502155522951!
>
>
>
From martin.senger at gmail.com Mon Jul 28 04:10:05 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Mon, 28 Jul 2008 12:10:05 +0800
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <488D4509.6070407@ucalgary.ca>
References:
<488B0123.3040608@bsc.es>
<4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
<4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
<488D25E6.7030601@ucalgary.ca>
<488D4509.6070407@ucalgary.ca>
Message-ID: <4d93f07c0807272110k66b54250q9373799210f28f22@mail.gmail.com>
> Perhaps we should make a statement of intendion before hammering out the
> details then ? ;-)
The first email in this thread says it (I believe):
Well, the first purpose (A) is obvious: we want to be able to deliver huge
> data from a service. So the service returns only a reference instead of the
> real data and the client can fetch the real data using some memory-friendly
> protocol (usualy a simple HTTP/GET).
>
> But it appeeared that it was not the only purpose. The second purpose (B)
> is to be able to send around already existing references (such as URLs of
> the EMBL or NCBI records).
>
Martin
--
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger
From soiland-reyes at cs.manchester.ac.uk Mon Jul 28 09:32:42 2008
From: soiland-reyes at cs.manchester.ac.uk (Stian Soiland-Reyes)
Date: Mon, 28 Jul 2008 10:32:42 +0100
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References:
<486CA2A7.4070702@bsc.es>
<2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com>
<48876ECA.1040109@cnio.es>
<2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com>
<488884CD.9030403@cnio.es>
<4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com>
<4889cf38.1aba720a.1b32.3a36@mx.google.com>
Message-ID:
Sorry if you got this email a couple of times, I ended up getting
bounces from moby-l@ even if I was posting to moby-dev@ - and changed
the wrong subscription!
On Mon, Jul 28, 2008 at 10:23 AM, Stian Soiland-Reyes
wrote:
> On Fri, Jul 25, 2008 at 2:03 PM, Edward Kawas wrote:
>> Don't worry about Taverna! Whatever we decide will most likely be the same
>> amount of work. So let's just get it right.
>
> Taverna 2 has built-in support for references (and their
> dereferencing) - so you could implement any referencing scheme you
> like, really, there's just a very simple interface to implement:
>
> http://taverna.cvs.sourceforge.net/taverna/t2reference/t2reference-api/src/main/java/net/sf/taverna/t2/reference/ExternalReferenceSPI.java?view=markup
>
>
> Example implementation:
>
> http://taverna.cvs.sourceforge.net/taverna/t2reference/t2reference-core-extensions/src/main/java/net/sf/taverna/t2/reference/impl/external/http/HttpReference.java?view=markup
>
>
> Plain HTTP references should work out of the box, though. You could
> make a version of this and add your own added-value semantics to the
> URIs - and it could still also work with other non-biomoby-services
> that recognised HTTP-URLs.
>
> t2 allows you to add translators between different reference schemes -
> so if you ended up with say "urn:biomoby:fish/1337" and you knew that
> you could download such references by using
> http://download.biomoby.org/fish/1337.xml - you can easily add a
> BiomobyReferenceToHttpReference translator that constructs such URIs
> given the biomoby reference.
>
>
> --
> Stian Soiland-Reyes, myGrid team
> School of Computer Science
> The University of Manchester
> http://www.mygrid.org.uk/dev/blog/
>
--
Stian Soiland-Reyes, myGrid team
School of Computer Science
The University of Manchester
http://www.mygrid.org.uk/dev/blog/
From jmfernandez at cnio.es Mon Jul 28 16:25:29 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Mon, 28 Jul 2008 18:25:29 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To:
References: <486CA2A7.4070702@bsc.es> <2A730847-8B4D-4ABE-9AED-23E3D68BA517@gmail.com> <48876ECA.1040109@cnio.es> <2D387E4C-AE3B-48D7-9121-925EB1281469@gmail.com> <488884CD.9030403@cnio.es> <4d93f07c0807241706t34b5a4c1w95049cd0f7728b55@mail.gmail.com> <2692C2D4-02A7-4B01-8A42-7BEF778B221E@gmail.com> <4889B5E8.6010903@mpiz-koeln.mpg.de> <4d93f07c0807250435g45dfb62cyf30b23f2de050986@mail.gmail.com>
Message-ID: <488DF2F9.5090207@cnio.es>
Last weeks we (at INB) were talking about binary objects and data-by-reference
in our internal list (mainly related to the usage of XInclude), and we found
that on one hand XInclude allows referencing non-XML content:
http://www.w3.org/TR/xinclude/#text-included-items
but that content must be string-like (with an associated encoding), not binary
one. So, when we realized that limitation we were talking about extending
XInclude in a non-standard way, considering a new parse style, called
'base64'. Standard XInclude understands parse='xml' and parse='text' styles,
and 'base64' parse mode would work like 'text' one, but encoding the content
in base64 before including it in the place of the reference.
What do you think about this hack?
Best Regards,
Jos? Mar?a
Mark Wilkinson wrote:
> On Fri, 25 Jul 2008 05:28:43 -0700, Pieter Neerincx
> wrote:
>
>
>
>> If you want to replace this with a reference the actual data would
>> still have to be a base64 encoded piece of text. So you can not make a
>> reference to an image that exists as a plain JPEG, GIF, etc. on a web
>> server.
>
> Martin and I discussed this while he was here and the behaviour of this
> case will be part of his RFC proposal. It seems to me that it would be
> silly to b64 encode something that is being passed by reference... but
> I agree that to do anything else is a 'hack'. Nevertheless, there are a
> lot of 'hacks' in Moby, and this one is one of the more sensible 'hacks'
> IMO...
>
> M
>
>
>
--
"There is no reason why anybody would want a computer in their home" -
Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???
"Premature optimization is the root of all evil." - Donald Knuth
Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3 Madrid (Spain)
**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.
From py at pingoured.fr Tue Jul 29 09:09:26 2008
From: py at pingoured.fr (Pierre-Yves Chibon)
Date: Tue, 29 Jul 2008 11:09:26 +0200
Subject: [MOBY-dev] php client for biomoby services
Message-ID: <488EDE46.5070804@pingoured.fr>
Dear list,
We have been trying to make a php client for a simple biomoby web
service that we developed few days ago.
We are facing a problem in the creation of the input XML. We think that
using php soap library we create soap XML and not moby XML. Which
obviously returns an error.
Has anyone of you ever try to make a simple client in php or just even
call a bioMoby service via php ?
Below is the input that is sent to our service and the XML retrieve from
that service
Thanks for your help,
Best regards,
Pierre
input:
a:1:{s:12:"Solstis_User";a:3:{s:8:"userName";s:4:"test";s:8:"password";s:4:"test";s:5:"email";s:12:"test at test.fr";}}
output:
<?xml version="1.0"
encoding="UTF-8"?>
<moby:MOBY xmlns:moby="http://www.biomoby.org/moby">
<moby:mobyContent>
<moby:serviceNotes>
<moby:mobyException moby:severity="error">
<moby:exceptionCode>600</moby:exceptionCode>
<moby:exceptionMessage>AN ERROR OCCURED DURING THE
SERVICE EXECUTION: Error in the XML input.
SystemID: null
Line (perhaps): 1
Column: 1</moby:exceptionMessage>
</moby:mobyException>
</moby:serviceNotes>
</moby:mobyContent>
</moby:MOBY>
From edward.kawas at gmail.com Tue Jul 29 14:56:04 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Tue, 29 Jul 2008 07:56:04 -0700
Subject: [MOBY-dev] php client for biomoby services
In-Reply-To: <488EDE46.5070804@pingoured.fr>
References: <488EDE46.5070804@pingoured.fr>
Message-ID: <488f2f88.2a528c0a.7d99.54ae@mx.google.com>
Hi Pierre,
Take a look at the following example that someone created at a TIGR
workshop.
https://biomoby.tigr.org/wiki/index.php/JCVI_Workshop#::_Step_5_-_how_to_cre
ate_a_MOBY_client_in_PHP_::
This example did work at the time and may still do the trick.
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Pierre-Yves Chibon
Sent: July-29-08 2:09 AM
To: MOBY-dev at lists.open-bio.org
Subject: [MOBY-dev] php client for biomoby services
Dear list,
We have been trying to make a php client for a simple biomoby web
service that we developed few days ago.
We are facing a problem in the creation of the input XML. We think that
using php soap library we create soap XML and not moby XML. Which
obviously returns an error.
Has anyone of you ever try to make a simple client in php or just even
call a bioMoby service via php ?
Below is the input that is sent to our service and the XML retrieve from
that service
Thanks for your help,
Best regards,
Pierre
input:
a:1:{s:12:"Solstis_User";a:3:{s:8:"userName";s:4:"test
";s:8:"password";s:4:"test";s:5:"email";s:12:"test at test.fr";}}
output:
<?xml version="1.0"
encoding="UTF-8"?>
<moby:MOBY xmlns:moby="http://www.biomoby.org/moby">
<moby:mobyContent>
<moby:serviceNotes>
<moby:mobyException moby:severity="error">
<moby:exceptionCode>600</moby:exceptionCode>
<moby:exceptionMessage>AN ERROR OCCURED DURING THE
SERVICE EXECUTION: Error in the XML input.
SystemID: null
Line (perhaps): 1
Column: 1</moby:exceptionMessage>
</moby:mobyException>
</moby:serviceNotes>
</moby:mobyContent>
</moby:MOBY>
<
/soapenv:Envelope>
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
From edward.kawas at gmail.com Tue Jul 29 15:23:58 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Tue, 29 Jul 2008 08:23:58 -0700
Subject: [MOBY-dev] php client for biomoby services
In-Reply-To: <488EDE46.5070804@pingoured.fr>
References: <488EDE46.5070804@pingoured.fr>
Message-ID: <488f3612.20d7720a.6190.1e4a@mx.google.com>
Just in case, you didn't get it working (I am sure that you did, but just in
case ...), I attached 2 php files that you can test with.
The test client is in the file 'moby_php_test_service.php' and the
moby_soap_factory.php needs to be either in the same directory as the test
service or on your PHP include path.
Thanks to Julian Tonti-Filippini for creating the PHP basis for the attached
examples!
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Pierre-Yves Chibon
Sent: July-29-08 2:09 AM
To: MOBY-dev at lists.open-bio.org
Subject: [MOBY-dev] php client for biomoby services
Dear list,
We have been trying to make a php client for a simple biomoby web
service that we developed few days ago.
We are facing a problem in the creation of the input XML. We think that
using php soap library we create soap XML and not moby XML. Which
obviously returns an error.
Has anyone of you ever try to make a simple client in php or just even
call a bioMoby service via php ?
Below is the input that is sent to our service and the XML retrieve from
that service
Thanks for your help,
Best regards,
Pierre
input:
a:1:{s:12:"Solstis_User";a:3:{s:8:"userName";s:4:"test
";s:8:"password";s:4:"test";s:5:"email";s:12:"test at test.fr";}}
output:
<?xml version="1.0"
encoding="UTF-8"?>
<moby:MOBY xmlns:moby="http://www.biomoby.org/moby">
<moby:mobyContent>
<moby:serviceNotes>
<moby:mobyException moby:severity="error">
<moby:exceptionCode>600</moby:exceptionCode>
<moby:exceptionMessage>AN ERROR OCCURED DURING THE
SERVICE EXECUTION: Error in the XML input.
SystemID: null
Line (perhaps): 1
Column: 1</moby:exceptionMessage>
</moby:mobyException>
</moby:serviceNotes>
</moby:mobyContent>
</moby:MOBY>
<
/soapenv:Envelope>
_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moby_php_test_service.php
Type: application/octet-stream
Size: 1308 bytes
Desc: not available
URL:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moby_soap_factory.php
Type: application/octet-stream
Size: 4300 bytes
Desc: not available
URL:
From groscurt at mpiz-koeln.mpg.de Wed Jul 30 08:51:58 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 30 Jul 2008 10:51:58 +0200
Subject: [MOBY-dev] [MOBY-l] Apologies,
a hint for MIPS-Service substitutions, and an invitation
In-Reply-To:
References:
Message-ID: <48902BAE.7000501@mpiz-koeln.mpg.de>
Mark wrote:
> This all brings me to my final comment - I currently have two
> summer-students who are working on the problem of automated workflow
> repair (i.e. finding equivalent services to fix broken workflows).
> We're going to be inviting everyone to participate in a Moby Service
> Annotation Jamboree later this summer. It will be a "virtual"
> jamboree - just come to the website. We'll have an interface there
> through which you can deeply annotate Moby services (inputs, outputs,
> underlying resources, service types, authorities, etc.) with the goal
> of improving service discovery. I encourage all Moby users to
> participate and improve the quality of our service annotations! At a
> minimum, I hope that the current service providers will visit the
> website and annotate their own services :-)
I was wondering how you store this information ? are they mapped to the
services/datatypes etc in the moby central database ?
Here at the mpiz we already thought about offering this feature as a
permanent service to everyone, so everyone can tag services/datatypes
etc. Have you ever considered such kind of things ? I recently modified
dashboard and my local moby database to have a prototype for service
tagging. Unfortunately i can only offer you some screenshots, but I was
wondering if such extension could be worth considering to integrate it
in the official moby api ? you can see the first idea at
http://bioinfo.mpiz-koeln.mpg.de/tagging2.jpg
so my suggestion is to think about a tagging system (via dashboard / web
portal), which is permant available and allows everyone to tag moby stuff ?!
best
andreas
--
=====================================================
Dipl. Bioinf. Andreas Groscurth
Bioinformatics Software Developer
Plant Computational Biology group
Max-Planck Institute for plant breeding research
Carl-von-Linne Weg 10
50829 Cologne
Germany
+49(0) 221 5062449
=====================================================
From pieter.neerincx at gmail.com Wed Jul 30 13:51:54 2008
From: pieter.neerincx at gmail.com (Pieter Neerincx)
Date: Wed, 30 Jul 2008 15:51:54 +0200
Subject: [MOBY-dev] data by reference - a request for comments
In-Reply-To: <4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
References:
<488B0123.3040608@bsc.es>
<4d93f07c0807260452g3048765ev6f84be49dc8cf26d@mail.gmail.com>
<4d93f07c0807271829v10ff50d5y2c6f83b279256fdb@mail.gmail.com>
Message-ID:
Hi Martin,
Had to think about the issues you raised below for a while...
On 28 Jul 2008, at 03:29, Martin Senger wrote:
> I am trying to figure out how I would implement the data by
> reference in
> order to achieve the main purpose - not no have all data in memory.
> I know
> how I would do it for the references being on the primitive types
> level, but
> it is less clear to me how I would do it with the references on other
> levels. Perhaps you can help me to explain how you would do it (or
> even
> Pieter can explore how he is already doing it)?
What I use mostly are http(s) references. I download them and save
them as tmp files in a tmp dir. What happens next completely depends
on the service.
Sometimes I'm lazy and simply use a DOM parser to load the entire
thing in memory for convenient access to all nodes. (Note that I'm
living in a Perl world and therefore this makes sense. Using a
reference prevents DOM parsing of the SOAP message by SOAP::Lite which
has an expansion factor 5-7 times worse than DOM parsing of only the
Moby payload.)
If I cannot afford to be lazy, because the Moby payload tends to be
big, I use hybrid parsers. I never use pure streaming (SAX) parsers.
Hybrid means I use a streaming parser to divide the input into certain
chunks/fragments. Next I loop over the chunks and load them completely
into memory. This way I only have one chunk at a time in memory plus a
few globals like for example a counter to keep track of the amount of
BLAST hits. This hybrid parsing can be either for XML or for legacy
data formats like BLAST reports in tabular format.
In one case I use an XSLT to parse and convert one type of Moby XML
message into an SVG (is also XML) image. The XSLT is pretty cool and
is also some kind of hybrid. Basically it works similar to a flat file
indexing system. You define certain nodes of interest. It parses the
XML (SAX not DOM) once to make an index of where these nodes occur in
the XML. This index is kept in memory, but that is a lot smaller since
it stores only where a certain node occurs not it's contents and it
stores only nodes you need and specified not everything. Next it
parses the XML again (SAX not DOM) to do your business logic.
The inconvenience of a pure streaming parser is that you cannot jump
around the data. Once the streaming has parsed and passed a certain
point it forgets about that data unless you decide to store it in
variables. So, if you parse 100 BLAST hits and when you parse number
88 you want to go back to compare it to number 12, you can not unless
you stored number 12 somehow. With the XSLT you don't have to store
it, as you could have indexed the BLAST hits. So, when you need 12
again the parser jumps back to that point and parses only the chunk/
fragment for number 12 again. This means some parts of the XML might
get parsed more than once, but for scalability that is much better
than parsing the entire XML several times or storing large pieces in
memory.
The disadvantage of XLSTs in my opinion is that it works radically
different than any language I've seen before. Perl and Java are
different, but they share many concepts like if, else, while, for
loops. Developing XSLTs requires a completely different way of
thinking and I never found it easy.
> For references on the primitive type level, I would do a usual
> parsing of a
> Moby message and when I find a reference, I would resolve it do a
> local file
> and I would pass to my service class the data as a file reference.
> It would
> be up to the service to read the file contents to the memory or not
> - that
> would depend what it needs to do with the data.
>
> However, on the higher level, such as mobyData level, the reference
> must be
> treated differently. If I do the same (resolving the reference to a
> local
> file), I still need to call a parser again to parse the contents of
> the
> local file. Because I still want to give my service already parsed
> data (and
> not a biomoby XML), and because I do not want to have all in memory,
> I need
> to create local files for each primitive type and do the same as above
> (passing my service local file references). I do not see much other
> choices.
Not only would you have to do that for the primitives. If you want to
provide pre-parsed data to a service you would have to do it for the
values of namespace and id attributes at any level of mobyData too
right? I assume we are talking about MoSeS here were based on the data
in BioMoby Central code is generated both to parse the Biomoby XML
inputs of a service as well as to compile its results as BioMoby XML.
Saving everything pre-parsed as tmp files to disk, doesn't make much
sense too me. If you would do it both for the entire message and for
all it's possible dissected parts it would cause quite some redundancy
and overhead.
But how to do it with a hybrid DOM/SAX parser or with an XSLT that
uses indices? You would have to generate the code to chunk the input
for a hybrid parser or you have to generate the code to index certain
nodes for the XSLT. This means you need to know how the business logic
of the services works to figure out at which level level of the XML
you would have to chunk or index. I know Mark has this famous slide
with the God of BioMoby, but I'm afraid even the almighty Martin
cannot predict based on the info in BioMoby Central how the business
logic of a services works :(... In the current MoSeS parsing of the
input and the business logic are two separate things, but if you want
to improve scalability with a hybrid SAX/DOM or XSLT parser, parsing
the input will become an essential part of the business logic. What
you could do is generate some disabled example code to chunk/index at
any level and then a developer could uncomment the lines required.
Furthermore there are some obvious levels at which chunking/indexing
makes sense. Take for example a huge Collection of Simples. It would
make sense to chunk/index at the level of the pseudo-root element of
the Simples. I don't think it's possible to pre-generate all code to
parse the input in the right order and with chunking/indexing at the
right levels, but with a few smart examples a developer should be able
to quickly change the order and/or adapt to chunking/indexing further
up or down the tree.
> Am I missing a point here?
No, making a typical "currency convertor" example web service with a
single small input and a single small output is easy, but making a web
service that scales well is really hard!
There is a Perl module in the BioMoby CVS for XSLT based parsing. It
was developed in France some time ago if I remember correctly and
AFAIK it is not maintained, so it probably lacks some of the newer
stuff like proper error handling, async services etc., but it might be
a nice example of what can be done with XSLTs.
Ok, those of you who managed to read all the way up till here earned a
beer on the next BioMoby developers meeting!
Cheers,
Pi
> The above is doable, of course. But you see why I wanted to have
> references
> only on the primitive type level. If we have them on any level (as
> it seems
> to be) I need, in my implementation, actually make another (local)
> references for the primitive type, anyway. Of course, it is better
> than the
> remote references because I do not need to make a network connection
> for
> each individual primitive type (point taken, Pieter) - but it still
> to be
> done.
>
> Cheers,
> Martin
>
> --
> Martin Senger
> email: martin.senger at gmail.com,m.senger at cgiar.org
> skype: martinsenger
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034
Dreijenlaan 3
6703 HA Wageningen
The Netherlands
phone: +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype: pieter.online
-------------------------------------------------------------
From markw at illuminae.com Wed Jul 30 14:54:23 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Wed, 30 Jul 2008 07:54:23 -0700
Subject: [MOBY-dev] [MOBY-l] Apologies,
a hint for MIPS-Service substitutions, and an invitation
In-Reply-To: <48902BAE.7000501@mpiz-koeln.mpg.de>
References:
<48902BAE.7000501@mpiz-koeln.mpg.de>
Message-ID:
On Wed, 30 Jul 2008 01:51:58 -0700, Andreas Groscurth
wrote:
> I was wondering how you store this information ? are they mapped to the
> services/datatypes etc in the moby central database ?
At the moment we're not entirely sure where to put it. I am loathe to put
it in Moby Central as we don't yet know what the quality of the annotation
will be (though I guess Moby Central isnt' curated either, so... it
probably wouldn't hurt). Moreover, the Moby Central API isn't able to
query at the level of detail that we will be capturing these annotations,
so there's no compelling reason to put it there. For now, we're putting
it in a local RDF store, but we may eventually move it into Freebase
(http://www.freebase.com/) to enable direct queries there. we'll see how
it goes.
> so my suggestion is to think about a tagging system (via dashboard / web
> portal), which is permant available and allows everyone to tag moby
> stuff ?!
Certainly I think we need a dashboard panel that contains the annotation
interface (called "ED" - the Entity Describer http://entitydescriber.org;
written by our very own Eddie Kawas :-) ). ED is currently a bit too
tightly linked to Connotea, but we're in the process of ripping it out and
making it more standalone for this Jamboree.
Eddie, what do you think of a dashboard ED panel?
M
--
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada
From edward.kawas at gmail.com Wed Jul 30 14:59:10 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 30 Jul 2008 07:59:10 -0700
Subject: [MOBY-dev] [MOBY-l] Apologies,
a hint for MIPS-Service substitutions, and an invitation
In-Reply-To:
References: <48902BAE.7000501@mpiz-koeln.mpg.de>
Message-ID: <489081c6.22bd720a.2434.ffffe4ae@mx.google.com>
I like the dashboard panel idea a lot. Seems like the best place to keep
that information. Nice thinking Andreas!
Eddie
-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Mark Wilkinson
Sent: July-30-08 7:54 AM
To: Core developer announcements; moby-l at lists.open-bio.org
Subject: Re: [MOBY-dev] [MOBY-l] Apologies, a hint for MIPS-Service
substitutions, and an invitation
On Wed, 30 Jul 2008 01:51:58 -0700, Andreas Groscurth