[MOBY-dev] Re: [MOBY] join operations

[Mark Wilkinson]

Hi Ken, 

A bit of a rambling response... as it is 10:00Pm on my last day of work
(Phil L. knows exactly what this means ;-) )

I'm c.c'ing this response to the moby-dev list because I think it jarrs
a lot of nerves that need to be jarred...

I have a big smile on my face right now... not because I have an answer,
but because I *don't* have an answer and wish I did (or more
importantly, insist that the final MOBY spec does!).;

At the moment MOBY handles *only* queries of the type:

(discover, and) select n from foreignservicen where value=x

I can't state strongly enough how crappy MOBY is at solving any more
complex problem than that!

I have submitted a grant proposal to study more complex boolean
queries... using set theory (ack!)...  but what we really need are
parameters in MOBY queries, and at the moment we don't have them.

Quite frankly, the problem of *representing* even simple biological
information and service type information in  our ontology is
sufficiently large to have commandeered most of our time :-(

I think the solution to parameter-based services (e.g. Blast)  perhaps
lies somewhere in the PISE project, but again, I haven't had time to
think through it fully - that might be complete bullshit.

Regardless, for the service you want to create, given the existing MOBY
system, it can't be done... you have to fudge it with combinations of
simple queries.

I'm sorry  :-(  One day we will have solved these problems, but so far
we are still infants and are still teething on the deeper complex
problems...
 
This leads to the bane of my life right now... MOBY is wonderful at
solving the most basic (and common) search and retrieve problem; however
under MOBY, today, you could legitimately create a service that takes
keywords as input and returns proteins with keyword-annotations that
also have knockouts as output... but what do you *call* that
service?!!?  What is the transformation type??  You are allowed (today)
to call it whatever you like - you can register the
"superknockoutkeywordservice" as a service type, and then register your
service as a service of that type... but who the hell knows what your
service actually *does* to the input data to get that output
datatype?!?!?

MOBY services (today) have the signature INPUT+TRANSFORM+OUTPUT, and
this is supposed to be sufficient for a client to identify the desired
service... but it is obviously not.  Your desired service is a perfect
example of that!  How do you describe the "transform" that you are
making?!?

I think, in parallel with the fantastic work that Damian and Andrew are
doing v.v service description and data transport technology, we need to
spend energy thinking deeply about service type description - I see this
as a critical problem (and I think that myGrid has a lot to teach us
about this!!).   I believe this will be a deadly problem for MOBY,
regardless of the data representation and/or transport layer that we
finally decide to use - at the end of the day the service description
must be both human readable and machine readable... ack!!!!   Damian
mentioned that he might have a solution to this today during the
conference call, and I would like to pursue this either on the list, or
in the calls, or even between the two of us personally (??) because this
is the big octopus on my brain right now and it is driving me nuts!... 
I can't sleep!!

Anyway, the bad news for you personally is that the service you want to
set up is a bugger... we can fudge it, but it will always be
unsatisfactory.

Regardless, I can help you fudge it!... or not.  Whatever you need :-) 
I'm just sure you will not be happy with your service until we have
clear solution to the service description problem.  I desperately want
us to find a good solution to this problem (perhaps before all other
problems we are dealing with?)

M



On Wed, 2003-01-22 at 20:17, Ken Steube wrote:
> Hey Mark, what are your thoughts on handling join operations?
> 
> To be specific I'll give an example:  Say I write a service that
> provides protein data given keywords and another service that tells
> which proteins have known knockouts.  I then want to find all the
> proteins which have both the keywords and knockouts.  So I run both
> services and perform a join on the results.
> 
> Since both services reside on one server it would be really beneficial
> to do the join on that server.  Is it possible to pass the two
> requests to some kind of "join service" that would run the two, join
> on the protein IDs, and return a joined object? This joined object
> would be a superclass of the first two.
> 
> Of course I could write a third service that takes args of keywords
> and a "require knockouts" flag but as I understand it that's not
> really MOBYesque.
> 
> Ken
> 
> -------------------------------------
> Ken Steube            steube at sdsc.edu
> San Diego Supercomputer Center @ UCSD
> San Diego, California             USA
> 
-- 
=======================================================================
                                    |--==\
Mark Wilkinson                       \==-|       1001010010010001001010
Bioinformatics Consultant             \=/        0010010010100101110010
Illuminae Media                       /-\        0010101110110100100101
727 6th Ave. N.                      /-==|       0010100100111101010010
Saskatoon, SK, Canada               |==-/        0101001000100101001011
S7K 2S8                              \=/         0100100100010010010101
+1 (306) 373 3841                     /\         1110101101110101001001
markw at illuminae.com                  /=-\        1101001010100101010101
                                    |--==\
=======================================================================