From kebil at psb.ugent.be  Tue Nov  4 08:02:49 2008
From: kebil at psb.ugent.be (Kenny Billiau)
Date: Tue, 4 Nov 2008 14:02:49 +0100 (CET)
Subject: [MOBY-dev] XML parsing takes forever
In-Reply-To: <490a1b7c.0fba720a.690e.4995@mx.google.com>
References: <Pine.LNX.4.64.0810301431300.27319@midas.psb.ugent.be>
	<4909c0c3.27b38c0a.1e6e.fffff90d@mx.google.com>
	<Pine.LNX.4.64.0810301541530.27319@midas.psb.ugent.be>
	<4909ff61.27f8720a.750d.76c6@mx.google.com>
	<Pine.LNX.4.64.0810301950170.27319@midas.psb.ugent.be>
	<490a03ee.22ba720a.17d1.ffff9006@mx.google.com>
	<490A10F9.5050408@cnio.es>
	<490a1b7c.0fba720a.690e.4995@mx.google.com>
Message-ID: <Pine.LNX.4.64.0811041125310.27319@midas.psb.ugent.be>

Hi,

I've run a little testprogram to see where most of the time is spent. I use a stripped down version of the xml file I've sent you. All but a few ElementContainers and no sequence. Just to keep the processing time low enough to debug.

      use MOSES::MOBY::Parser;

      open(FILE, '<', $ARGV[0]);
      my @lines = <FILE>;
      close FILE;

      my $parser = MOSES::MOBY::Parser->new();
      print $parser->parse( method => 'string', data => join(q{}, @lines) );


[davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
[davebowman at hal2000 ~]$ dprofpp
Exporter::Heavy::heavy_export has 14 unstacked calls in outer
Config::Simple::import_from has 1 unstacked calls in outer
utf8::AUTOLOAD has -1 unstacked calls in outer
Config::Simple::verbose has 1 unstacked calls in outer
Config::Simple::vars has 1 unstacked calls in outer
AutoLoader::AUTOLOAD has -3 unstacked calls in outer
Fcntl::AUTOLOAD has -6 unstacked calls in outer
Fcntl::__ANON__ has 6 unstacked calls in outer
Exporter::export has -14 unstacked calls in outer
utf8::SWASHNEW has 1 unstacked calls in outer
Total Elapsed Time = 8.470220 Seconds
   User+System Time = 7.750220 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  68.1   5.280  5.280  69736   0.0001 0.0001  IO::Scalar::print
  6.94   0.538  0.783  64088   0.0000 0.0000  Dumpvalue::stringify
  6.04   0.468  7.173  33755   0.0000 0.0002  Dumpvalue::unwrap
  4.30   0.333  0.333  47091   0.0000 0.0000  File::Spec::Unix::canonpath
  2.43   0.188  7.157  33006   0.0000 0.0002  Dumpvalue::DumpElem
  2.13   0.165  0.165  59776   0.0000 0.0000  overload::mycan
  1.78   0.138  0.224  15990   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
  1.42   0.110  0.708    539   0.0002 0.0013  MOSES::MOBY::Cache::Central::_createDataTypeFromXML
  1.39   0.108  0.108 142023   0.0000 0.0000  UNIVERSAL::isa
  1.29   0.100  0.100    208   0.0005 0.0005  MOSES::MOBY::Cache::Registries::init
  1.20   0.093  0.357  17359   0.0000 0.0000  overload::OverloadedStringify
  1.16   0.090  0.090  30407   0.0000 0.0000  XML::LibXML::Node::nodeName
  1.02   0.079  0.079    539   0.0001 0.0001  MOSES::MOBY::Cache::Central::_clean
  0.99   0.077  0.418    207   0.0004 0.0020  MOSES::MOBY::Generators::Utils::find_file
  0.94   0.073  0.203   3147   0.0000 0.0001  XML::LibXML::Element::getChildrenByTagName

It's fairly obivous now ;)

If I remove the overload operator in MOSES::MOBY::Base, time is reduced 
15fold! But for the 250KB file I've sent, it still takes us a minute to 
parse.

If I simply parse the file with XML::LibXML and no callbacks or custom 
handlers, it's parsed instantly:

[davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
[kebil at psbdev01 ~]$ dprofpp
Total Elapsed Time = 0.079034 Seconds
   User+System Time = 0.079034 Seconds


-Kenny, slowly getting to the bottom of this ;)


On Thu, 30 Oct 2008, Edward Kawas wrote:

> Now I can confirm that it is slow parsing and not an infinite loop! The XML
> file took over an hour on my virtual machine to parse. I will see what I can
> do with other SAX parsers for Perl.
>
> Thanks,
>
> Eddie
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Jos? Mar?a
> Fern?ndez Gonz?lez
> Sent: October-30-08 12:55 PM
> To: Core developer announcements
> Subject: Re: [MOBY-dev] XML parsing takes forever
>
> Hi everybody,
> 	I'm using the XML::LibXML SAX parser for other projects and tasks,
> and it can
> be really, really slow. You don't realize that until you are parsing a large
> file from a pipe, and you see that most of the time the program which is
> parsing is CPU (and not I/O) bounded.
>
> 	Best Regards,
> 		Jos? Mar?a
>
> Edward Kawas wrote:
>> I think that you are right about the parser not going into an infinite
> loop.
>> For some reason, the SAX parser seems to be really slow (although, I don't
>> want to say that until I know for sure).
>>
>> Eddie
>>
>> -----Original Message-----
>> From: moby-dev-bounces at lists.open-bio.org
>> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
>> Sent: October-30-08 11:56 AM
>> To: Core developer announcements
>> Subject: Re: [MOBY-dev] XML parsing takes forever
>>
>> Hi,
>>
>> thx for helping to investigate the problem!
>>
>> I'm pretty sure the parser gets out of it's mess. When you truncate the
>> XML file by removing all but a few ElementContainers, it takes awhile, but
>
>> it actually returns without error.
>>
>> And memory consuption doesn't seem to be a problem either, as mostly 'top'
>
>> reveals that the invoked webservice script uses a steady amount of memory.
>>
>> thx,
>> Kenny
>>
>> On Thu, 30 Oct 2008, Edward Kawas wrote:
>>
>>> So this one might take some time... from what I can see, our parser goes
>>> into an infinite loop (although, I am not certain about that ...).
>>>
>>> I will play around with this XML file and get back to you.
>>>
>>> Thanks,
>>>
>>> Eddie
>>>
>>> -----Original Message-----
>>> From: moby-dev-bounces at lists.open-bio.org
>>> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
>>> Sent: October-30-08 8:03 AM
>>> To: Core developer announcements
>>> Subject: Re: [MOBY-dev] XML parsing takes forever
>>>
>>> Hi,
>>>
>>> here you go,
>>>
>>> Kenny
>>>
>>> On Thu, 30 Oct 2008, Edward Kawas wrote:
>>>
>>>> Hi,
>>>>
>>>> This is the right list! Could you send me the XML for your second
>> service?
>>>> Thanks,
>>>>
>>>> Eddie
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>
>>
>
>

-- 

From kenny.billiau at psb.ugent.be  Tue Nov  4 08:03:47 2008
From: kenny.billiau at psb.ugent.be (Kenny Billiau)
Date: Tue, 4 Nov 2008 14:03:47 +0100 (CET)
Subject: [MOBY-dev] XML parsing takes forever
Message-ID: <Pine.LNX.4.64.0811041403160.27319@midas.psb.ugent.be>

Hi,

I've run a little testprogram to see where most of the time is spent. I use a 
stripped down version of the xml file I've sent you. All but a few 
ElementContainers and no sequence. Just to keep the processing time low enough 
to debug.

      use MOSES::MOBY::Parser;

       open(FILE, '<', $ARGV[0]);
       my @lines = <FILE>;
       close FILE;

       my $parser = MOSES::MOBY::Parser->new();
       print $parser->parse( method => 'string', data => join(q{}, @lines) );


[davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
[davebowman at hal2000 ~]$ dprofpp
Exporter::Heavy::heavy_export has 14 unstacked calls in outer
Config::Simple::import_from has 1 unstacked calls in outer
utf8::AUTOLOAD has -1 unstacked calls in outer
Config::Simple:: verbose has 1 unstacked calls in outer
Config::Simple:: vars has 1 unstacked calls in outer
AutoLoader::AUTOLOAD has -3 unstacked calls in outer
Fcntl:: AUTOLOAD has -6 unstacked calls in outer
Fcntl: :__ANON__ has 6 unstacked calls in outer
Exporter::export has -14 unstacked calls in outer
utf8::SWASHNEW has 1 unstacked calls in outer
Total Elapsed Time = 8.470220 Seconds
   User+System Time = 7.750220 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
   68.1   5.280  5.280  69736   0.0001 0.0001  IO::Scalar::print
   6.94   0.538  0.783  64088   0.0000 0.0000  Dumpvalue::stringify
   6.04   0.468  7.173  33755   0.0000 0.0002  Dumpvalue::unwrap
   4.30   0.333  0.333  47091   0.0000 0.0000  File::Spec::Unix::canonpath
   2.43   0.188  7.157  33006   0.0000 0.0002  Dumpvalue::DumpElem
   2.13   0.165  0.165  59776   0.0000 0.0000  overload::mycan
   1.78   0.138  0.224  15990   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
   1.42   0.110  0.708    539   0.0002 0.0013
   MOSES::MOBY::Cache::Central::_createDataTypeFromXML
   1.39   0.108  0.108 142023   0.0000 0.0000  UNIVERSAL::isa
   1.29   0.100  0.100    208   0.0005 0.0005
   MOSES::MOBY::Cache::Registries::init
   1.20   0.093  0.357  17359   0.0000 0.0000  overload::OverloadedStringify
   1.16   0.090  0.090  30407   0.0000 0.0000  XML::LibXML::Node::nodeName
   1.02   0.079  0.079    539   0.0001 0.0001
   MOSES::MOBY::Cache::Central::_clean
   0.99   0.077  0.418    207   0.0004 0.0020
   MOSES::MOBY::Generators::Utils::find_file
   0.94   0.073  0.203   3147   0.0000 0.0001
   XML::LibXML::Element::getChildrenByTagName

It's fairly obivous now ;)

If I remove the overload operator in MOSES::MOBY::Base, time is reduced 15fold! 
But for the 250KB file I've sent, it still takes us a minute to parse.

If I simply parse the file with XML::LibXML and no callbacks or custom 
handlers, it's parsed instantly:

[davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
[kebil at psbdev01 ~]$ dprofpp
Total Elapsed Time = 0.079034 Seconds
    User+System Time = 0.079034 Seconds


-Kenny, slowly getting to the bottom of this ;)


On Thu, 30 Oct 2008, Edward Kawas wrote:

>  Now I can confirm that it is slow parsing and not an infinite loop! The XML
>  file took over an hour on my virtual machine to parse. I will see what I can
>  do with other SAX parsers for Perl.
>
>  Thanks,
>
>  Eddie
>
>  -----Original Message-----
>  From: moby-dev-bounces at lists.open-bio.org
>  [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Jos? Mar?a
>  Fern?ndez Gonz?lez
>  Sent: October-30-08 12:55 PM
>  To: Core developer announcements
>  Subject: Re: [MOBY-dev] XML parsing takes forever
>
>  Hi everybody,
>  	I'm using the XML::LibXML SAX parser for other projects and tasks,
>  and it can
>  be really, really slow. You don't realize that until you are parsing a large
>  file from a pipe, and you see that most of the time the program which is
>  parsing is CPU (and not I/O) bounded.
>
>   Best Regards,
>    Jos? Mar?a
>
>  Edward Kawas wrote:
> >  I think that you are right about the parser not going into an infinite
>  loop.
> >  For some reason, the SAX parser seems to be really slow (although, I don't
> >  want to say that until I know for sure).
> > 
> >  Eddie
> > 
> >  -----Original Message-----
> >  From: moby-dev-bounces at lists.open-bio.org
> >  [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
> >  Sent: October-30-08 11:56 AM
> >  To: Core developer announcements
> >  Subject: Re: [MOBY-dev] XML parsing takes forever
> > 
> >  Hi,
> > 
> >  thx for helping to investigate the problem!
> > 
> >  I'm pretty sure the parser gets out of it's mess. When you truncate the
> >  XML file by removing all but a few ElementContainers, it takes awhile, but
> 
> >  it actually returns without error.
> > 
> >  And memory consuption doesn't seem to be a problem either, as mostly 'top'
> 
> >  reveals that the invoked webservice script uses a steady amount of memory.
> > 
> >  thx,
> >  Kenny
> > 
> >  On Thu, 30 Oct 2008, Edward Kawas wrote:
> > 
> > >  So this one might take some time... from what I can see, our parser goes
> > >  into an infinite loop (although, I am not certain about that ...).
> > > 
> > >  I will play around with this XML file and get back to you.
> > > 
> > >  Thanks,
> > > 
> > >  Eddie
> > > 
> > >  -----Original Message-----
> > >  From: moby-dev-bounces at lists.open-bio.org
> > >  [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
> > >  Sent: October-30-08 8:03 AM
> > >  To: Core developer announcements
> > >  Subject: Re: [MOBY-dev] XML parsing takes forever
> > > 
> > >  Hi,
> > > 
> > >  here you go,
> > > 
> > >  Kenny
> > > 
> > >  On Thu, 30 Oct 2008, Edward Kawas wrote:
> > > 
> > > >  Hi,
> > > > 
> > > >  This is the right list! Could you send me the XML for your second
> >  service?
> > > >  Thanks,
> > > > 
> > > >  Eddie
>>> _______________________________________________
> > >  MOBY-dev mailing list
> > >  MOBY-dev at lists.open-bio.org
> > >  http://lists.open-bio.org/mailman/listinfo/moby-dev
> > > 
> > 
> 
>

-- 

From kebil at psb.ugent.be  Wed Nov  5 10:57:00 2008
From: kebil at psb.ugent.be (Kenny Billiau)
Date: Wed, 5 Nov 2008 16:57:00 +0100 (CET)
Subject: [MOBY-dev] XML parsing takes forever
In-Reply-To: <Pine.LNX.4.64.0811041403160.27319@midas.psb.ugent.be>
References: <Pine.LNX.4.64.0811041403160.27319@midas.psb.ugent.be>
Message-ID: <Pine.LNX.4.64.0811051418290.27319@midas.psb.ugent.be>

Hi,

after some more tests, I've succesfully reduced the parsing time from over 
1 hour 15 minutes to 17 seconds. Which is not ideal yet, but at least workable :)

This is the profiling result for the original environment. The test xml file is the original file I've sent here without the sequence.

[davebowman at hal2000 ]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_no_dna.xml
Trying to parse.. Done.
[davebowman at hal2000 ]$ dprofpp
Total Elapsed Time = 4348.764 Seconds
   User+System Time = 4344.374 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  97.3   4227. 4227.2 399336   0.0011 0.0011  IO::Scalar::print
  0.88   38.06 4357.4 189130   0.0000 0.0023  Dumpvalue::unwrap
  0.80   34.54 51.532 369048   0.0000 0.0000  Dumpvalue::stringify
  0.54   23.63 23.634 463568   0.0000 0.0000  File::Spec::Unix::canonpath
  0.33   14.23 4356.7 188364   0.0000 0.0023  Dumpvalue::DumpElem


First I simply removed the overload operator "" (as_string). I still get the same results and processing time is reduced drastically.
I don't really know why this way of stringifying (complex) perl types is used here. 
Is there a special reason for it? Can't we simply use Data::Dumper (which prints to a $var by default)?

[davebowman at hal2000 ]$ dprofpp
Total Elapsed Time = 35.85114 Seconds
   User+System Time = 33.54114 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  64.5   21.64 21.646 463568   0.0000 0.0000  File::Spec::Unix::canonpath
  22.9   7.698 31.602 231569   0.0000 0.0000  File::Spec::Unix::catfile
  19.7   6.607  6.607   2144   0.0031 0.0031  MOSES::MOBY::Cache::Registries::init
  18.6   6.251 37.772   2143   0.0029 0.0176  MOSES::MOBY::Generators::Utils::find_file
  6.80   2.282 13.448 231569   0.0000 0.0000  File::Spec::Unix::catdir

A lot of calls are made to MOSES::MOBY::Generators::Utils::find_file, 
which in it's turn does a lot of calls with File::Spec. So I simply cache the results.
Time is reduced to 17 seconds:

Total Elapsed Time = 17.34032 Seconds
   User+System Time = 13.20032 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  51.5   6.807  6.807   2144   0.0032 0.0032  MOSES::MOBY::Cache::Registries::init
  9.01   1.190  2.005 162858   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
  4.65   0.614  0.614 310751   0.0000 0.0000  XML::LibXML::Node::nodeName
  4.64   0.612  1.655  32171   0.0000 0.0001  XML::LibXML::Element::getChildrenByTagName
  4.48   0.592  0.592   5515   0.0001 0.0001  MOSES::MOBY::Cache::Central::_clean

More then half of the time still goes to building up the MOBY environment for each processed element in the XML file. Isn't this a bit overkill? ;)
Is there a way to make a MOSES::MOBY factory that can handle the building up of the moby environment just once in the same (mod_perl) call? Or is there a reason this has been avoided?

I've included the patches for the result caching in 
MOSES::MOBY::Generators::Utils::find_file . there's not much to it. My 
version of MOSES::MOBY still is the one uploaded to cpan: Module MOSES::MOBY (E/EK/EKAWAS/MOSES-MOBY-0.86.tar.gz)

happy hacking,
Kenny


On Tue, 4 Nov 2008, Kenny Billiau wrote:

> Hi,
>
> I've run a little testprogram to see where most of the time is spent. I use a 
> stripped down version of the xml file I've sent you. All but a few 
> ElementContainers and no sequence. Just to keep the processing time low 
> enough to debug.
>
>     use MOSES::MOBY::Parser;
>
>       open(FILE, '<', $ARGV[0]);
>       my @lines = <FILE>;
>       close FILE;
>
>       my $parser = MOSES::MOBY::Parser->new();
>       print $parser->parse( method => 'string', data => join(q{}, @lines) );
>
>
> [davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
> [davebowman at hal2000 ~]$ dprofpp
> Exporter::Heavy::heavy_export has 14 unstacked calls in outer
> Config::Simple::import_from has 1 unstacked calls in outer
> utf8::AUTOLOAD has -1 unstacked calls in outer
> Config::Simple::  verbose has 1 unstacked calls in outer
> Config::Simple::  vars has 1 unstacked calls in outer
> AutoLoader::AUTOLOAD has -3 unstacked calls in outer
> Fcntl::  AUTOLOAD has -6 unstacked calls in outer
> Fcntl: :__ANON__ has 6 unstacked calls in outer
> Exporter::export has -14 unstacked calls in outer
> utf8::SWASHNEW has 1 unstacked calls in outer
> Total Elapsed Time = 8.470220 Seconds
>  User+System Time = 7.750220 Seconds
> Exclusive Times
> %Time ExclSec CumulS #Calls sec/call Csec/c  Name
>   68.1   5.280  5.280  69736   0.0001 0.0001  IO::Scalar::print
>   6.94   0.538  0.783  64088   0.0000 0.0000  Dumpvalue::stringify
>   6.04   0.468  7.173  33755   0.0000 0.0002  Dumpvalue::unwrap
>   4.30   0.333  0.333  47091   0.0000 0.0000  File::Spec::Unix::canonpath
>   2.43   0.188  7.157  33006   0.0000 0.0002  Dumpvalue::DumpElem
>   2.13   0.165  0.165  59776   0.0000 0.0000  overload::mycan
>   1.78   0.138  0.224  15990   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
>   1.42   0.110  0.708    539   0.0002 0.0013
>   MOSES::MOBY::Cache::Central::_createDataTypeFromXML
>   1.39   0.108  0.108 142023   0.0000 0.0000  UNIVERSAL::isa
>   1.29   0.100  0.100    208   0.0005 0.0005
>   MOSES::MOBY::Cache::Registries::init
>   1.20   0.093  0.357  17359   0.0000 0.0000  overload::OverloadedStringify
>   1.16   0.090  0.090  30407   0.0000 0.0000  XML::LibXML::Node::nodeName
>   1.02   0.079  0.079    539   0.0001 0.0001
>   MOSES::MOBY::Cache::Central::_clean
>   0.99   0.077  0.418    207   0.0004 0.0020
>   MOSES::MOBY::Generators::Utils::find_file
>   0.94   0.073  0.203   3147   0.0000 0.0001
>   XML::LibXML::Element::getChildrenByTagName
>
> It's fairly obivous now ;)
>
> If I remove the overload operator in MOSES::MOBY::Base, time is reduced 
> 15fold! But for the 250KB file I've sent, it still takes us a minute to 
> parse.
>
> If I simply parse the file with XML::LibXML and no callbacks or custom 
> handlers, it's parsed instantly:
>
> [davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
> [kebil at psbdev01 ~]$ dprofpp
> Total Elapsed Time = 0.079034 Seconds
>    User+System Time = 0.079034 Seconds
>
>
> -Kenny, slowly getting to the bottom of this ;)
>
>
> On Thu, 30 Oct 2008, Edward Kawas wrote:
>
>>   Now I can confirm that it is slow parsing and not an infinite loop! The
>>   XML
>>   file took over an hour on my virtual machine to parse. I will see what I
>>   can
>>   do with other SAX parsers for Perl.
>>
>>   Thanks,
>>
>>   Eddie
>>
>>   -----Original Message-----
>>   From: moby-dev-bounces at lists.open-bio.org
>>   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Jos? Mar?a
>>   Fern?ndez Gonz?lez
>>   Sent: October-30-08 12:55 PM
>>   To: Core developer announcements
>>   Subject: Re: [MOBY-dev] XML parsing takes forever
>>
>>   Hi everybody,
>>   	I'm using the XML::LibXML SAX parser for other projects and tasks,
>>   and it can
>>   be really, really slow. You don't realize that until you are parsing a
>>   large
>>   file from a pipe, and you see that most of the time the program which is
>>   parsing is CPU (and not I/O) bounded.
>>
>>    Best Regards,
>>     Jos? Mar?a
>>
>>   Edward Kawas wrote:
>> >   I think that you are right about the parser not going into an infinite
>>   loop.
>> >   For some reason, the SAX parser seems to be really slow (although, I 
>> >   don't
>> >   want to say that until I know for sure).
>> > 
>> >   Eddie
>> > 
>> >   -----Original Message-----
>> >   From: moby-dev-bounces at lists.open-bio.org
>> >   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
>> >   Sent: October-30-08 11:56 AM
>> >   To: Core developer announcements
>> >   Subject: Re: [MOBY-dev] XML parsing takes forever
>> > 
>> >   Hi,
>> > 
>> >   thx for helping to investigate the problem!
>> > 
>> >   I'm pretty sure the parser gets out of it's mess. When you truncate the
>> >   XML file by removing all but a few ElementContainers, it takes awhile, 
>> >   but
>> 
>> >   it actually returns without error.
>> > 
>> >   And memory consuption doesn't seem to be a problem either, as mostly 
>> >   'top'
>> 
>> >   reveals that the invoked webservice script uses a steady amount of 
>> >   memory.
>> > 
>> >   thx,
>> >   Kenny
>> > 
>> >   On Thu, 30 Oct 2008, Edward Kawas wrote:
>> > 
>> > >   So this one might take some time... from what I can see, our parser 
>> > >   goes
>> > >   into an infinite loop (although, I am not certain about that ...).
>> > > 
>> > >   I will play around with this XML file and get back to you.
>> > > 
>> > >   Thanks,
>> > > 
>> > >   Eddie
>> > > 
>> > >   -----Original Message-----
>> > >   From: moby-dev-bounces at lists.open-bio.org
>> > >   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny 
>> > >   Billiau
>> > >   Sent: October-30-08 8:03 AM
>> > >   To: Core developer announcements
>> > >   Subject: Re: [MOBY-dev] XML parsing takes forever
>> > > 
>> > >   Hi,
>> > > 
>> > >   here you go,
>> > > 
>> > >   Kenny
>> > > 
>> > >   On Thu, 30 Oct 2008, Edward Kawas wrote:
>> > > 
>> > > >   Hi,
>> > > > 
>> > > >   This is the right list! Could you send me the XML for your second
>> >   service?
>> > > >   Thanks,
>> > > > 
>> > > >   Eddie
>> > >  _______________________________________________
>> > >   MOBY-dev mailing list
>> > >   MOBY-dev at lists.open-bio.org
>> > >   http://lists.open-bio.org/mailman/listinfo/moby-dev
>> > > 
>> >
>> 
>> 
>
>

-- 
-------------- next part --------------
--- /usr/lib/perl5/site_perl/5.8.5/MOSES/MOBY/Generators/Utils.pm	2008-04-29 21:45:41.000000000 +0200
+++ ../modules/MOSES/MOBY/Generators/Utils.pm	2008-11-05 16:47:23.000000000 +0100
@@ -26,7 +26,7 @@
  # find a file located somewhere in @INC
  use MOSES::MOBY::Generators::Utils;
  my $file = MOSES::MOBY::Generators::Utils->find_file ('resource.file');
- 
+
 =cut
 
 =head1 DESCRIPTION
@@ -61,17 +61,29 @@
 
 =cut
 
+my %full_path_of = ();
+
 sub find_file {
     my ($self, $default_start, @names) = @_;
     my $fixed_part = File::Spec->catfile (@names);
+    return $full_path_of{ $fixed_part } if exists $full_path_of{ $fixed_part };
+
     my $result = File::Spec->catfile ($default_start, $fixed_part);
-    return $result if -e $result;
+    if (-e $result) {
+        $full_path_of{ $fixed_part } = $result;
+        return $result;
+    }
 
     foreach my $idx (0 .. $#INC) {
-	$result = File::Spec->catfile ($INC[$idx], $fixed_part);
-	return $result if -e $result;
+        $result = File::Spec->catfile ($INC[$idx], $fixed_part);
+        if (-e $result) {
+            $full_path_of{ $fixed_part } = $result;
+            return $result;
+        }
     }
-    return File::Spec->catfile ($default_start, $fixed_part);
+    $result = File::Spec->catfile ($default_start, $fixed_part);
+    $full_path_of{ $fixed_part } = $result;
+    return $result;
 }
 
 1;

From dmitry.repchevski at bsc.es  Thu Nov  6 15:58:07 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 06 Nov 2008 12:58:07 -0800
Subject: [MOBY-dev] INB Base64 (binary) type proposal
Message-ID: <49135A5F.20703@bsc.es>

Hello everybody,

These days there were a technical INB meeting in Malaga (Spain) and I 
like to comment some decisions taken there.

Most interesting to me is the introduction of new Moby primitive type 
"Binary" that will correspond to the xml schema:

********************************************************
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:tns="http://www.biomoby.org/moby" 
xmlns="http://www.biomoby.org/moby" 
xmlns:moby="http://www.biomoby.org/moby" 
xmlns:xs="http://www.w3.org/2001/XMLSchema" 
attributeFormDefault="qualified" elementFormDefault="qualified" 
version="1.0" targetNamespace="http://www.biomoby.org/moby">

<xs:element name="Binary" type="MobyBinary"></xs:element>

<xs:complexType name="MobyBinary">
<xs:simpleContent>
 <xs:extension
   base="xs:base64Binary"
   ns1:expectedContentTypes="application/octet-stream"
   xmlns:ns1="http://www.w3.org/2005/05/xmlmime">
 </xs:extension>
</xs:simpleContent>
</xs:complexType>

</xs:schema>
********************************************************

The encoding inside a Moby Message will be a standard base64 coded chunk 
of data:
********************************************************
<moby:Binary id="" namespace="">dGVzdA==</moby:Binary>
********************************************************

The need of new type is enough obvious: we need to pass binary data. 
Today, instead of using a primitive type many service developers mess a 
datatype tree with their own solutions (text-base64, b64_encoded_ppm, 
Zip_encoded, Image_PNG, Image_Encoded, glycoPNG ...)
Even we do not expect that service providers will change their services 
to utilize this new type, it would be very useful for many new types.

At the moment the encoding itself is a responsibility of a developer and 
such a type will free them from the need to encode/decode process.
The implementation is pretty easy and as soon as our key Perl developer 
is back from vacation we expect him to introduce it into the perl moby 
code.

So far this is the official INB proposal and we would be happy to here 
comments from other develops.
Any comments or suggestions are welcome.

Warm regards,

Dmitry

P.S. http://inb.bsc.es/documents/base64bit.pdf


From groscurt at mpiz-koeln.mpg.de  Thu Nov  6 07:45:54 2008
From: groscurt at mpiz-koeln.mpg.de (groscurt)
Date: Thu, 6 Nov 2008 13:45:54 +0100
Subject: [MOBY-dev] INB Base64 (binary) type proposal
In-Reply-To: <49135A5F.20703@bsc.es>
Message-ID: <0bf412a6235d581f18ff007cd76e92e6@mpizmail.mpiz-koeln.mpg.de>

> 
> P.S. http://inb.bsc.es/documents/base64bit.pdf

Object not found for me...

cheers
andreas




From dmitry.repchevski at bsc.es  Thu Nov  6 17:12:01 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 06 Nov 2008 14:12:01 -0800
Subject: [MOBY-dev] INB Base64 (binary) type proposal
In-Reply-To: <0bf412a6235d581f18ff007cd76e92e6@mpizmail.mpiz-koeln.mpg.de>
References: <0bf412a6235d581f18ff007cd76e92e6@mpizmail.mpiz-koeln.mpg.de>
Message-ID: <49136BB1.90604@bsc.es>

Opps I'm sorry...

http://inb.bsc.es/documents/Base64bit.pdf

uppercase...



From dcicinsain at mmb.pcb.ub.es  Thu Nov  6 08:26:23 2008
From: dcicinsain at mmb.pcb.ub.es (Damjan Cicin-Sain)
Date: Thu, 06 Nov 2008 14:26:23 +0100
Subject: [MOBY-dev] INB Base64 (binary) type proposal
In-Reply-To: <0bf412a6235d581f18ff007cd76e92e6@mpizmail.mpiz-koeln.mpg.de>
References: <0bf412a6235d581f18ff007cd76e92e6@mpizmail.mpiz-koeln.mpg.de>
Message-ID: <1225977983.6568.2.camel@damjan-laptop>

http://inb.bsc.es/documents/Base64bit.pdf

On Thu, 2008-11-06 at 13:45 +0100, groscurt wrote:
> > 
> > P.S. http://inb.bsc.es/documents/base64bit.pdf
> 
> Object not found for me...
> 
> cheers
> andreas
> 
> 
> 
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev


From py at pingoured.fr  Thu Nov  6 11:28:34 2008
From: py at pingoured.fr (Pierre-Yves Chibon)
Date: Thu, 06 Nov 2008 17:28:34 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <48528447.7040807@toulouse.inra.fr>
References: <4850BC9D.8090800@mpiz-koeln.mpg.de>	<485127d8.25bb720a.622e.0d84@mx.google.com>	<48512862.7030104@mpiz-koeln.mpg.de>	<4851431d.1420720a.52bf.243d@mx.google.com>	<48527CD8.9060904@mpiz-koeln.mpg.de>
	<48528447.7040807@toulouse.inra.fr>
Message-ID: <49131B32.2010605@pingoured.fr>

Sebastien Carrere wrote:
> Here is my list of Dead services (Ihave got also list for the 3 past 
> days for testRegistry and Inab registry) :
> http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral


Just by curiosity :
  - do you have a list of working services ?
  - do you have available your results from the testing repository ? (by 
the way, why are you testing it ?)

Thanks in advance,

Regards,

Pierre

From Sebastien.Carrere at toulouse.inra.fr  Thu Nov  6 12:16:47 2008
From: Sebastien.Carrere at toulouse.inra.fr (Sebastien Carrere)
Date: Thu, 06 Nov 2008 18:16:47 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49131B32.2010605@pingoured.fr>
References: <4850BC9D.8090800@mpiz-koeln.mpg.de>	<485127d8.25bb720a.622e.0d84@mx.google.com>	<48512862.7030104@mpiz-koeln.mpg.de>	<4851431d.1420720a.52bf.243d@mx.google.com>	<48527CD8.9060904@mpiz-koeln.mpg.de>	<48528447.7040807@toulouse.inra.fr>
	<49131B32.2010605@pingoured.fr>
Message-ID: <4913267F.8030307@toulouse.inra.fr>

Bonjour
>
> Just by curiosity :
>  - do you have a list of working services ?
Yes, I have:
For produciton registry:
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/list.mobycentral
For testing registry:
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/list.opencentral
For Inab registry:
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/list.Inab
>  - do you have available your results from the testing repository ? 
> (by the way, why are you testing it ?)
Here is the black list for testing registry:
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.opencentral

I test all these registries to filter really available services in 
Remora (you can choose the registry you want among these 3 - I can add 
others if needed).
By default, production registry is used.

Sebastien



-------------- next part --------------
A non-text attachment was scrubbed...
Name: Sebastien_Carrere.vcf
Type: text/x-vcard
Size: 387 bytes
Desc: not available
URL: <http://lists.open-bio.org/pipermail/moby-dev/attachments/20081106/fac42ea4/attachment.vcf>

From serr at ac.uma.es  Thu Nov  6 12:14:00 2008
From: serr at ac.uma.es (Sergio Ramirez Ramirez)
Date: Thu, 06 Nov 2008 18:14:00 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49131B32.2010605@pingoured.fr>
References: <4850BC9D.8090800@mpiz-koeln.mpg.de>	<485127d8.25bb720a.622e.0d84@mx.google.com>	<48512862.7030104@mpiz-koeln.mpg.de>	<4851431d.1420720a.52bf.243d@mx.google.com>	<48527CD8.9060904@mpiz-koeln.mpg.de>	<48528447.7040807@toulouse.inra.fr>
	<49131B32.2010605@pingoured.fr>
Message-ID: <491325D8.5050402@ac.uma.es>

Hello Pierre.

Just in advance. As Dimitry has comment we had a meeting of the INB this 
week, and one of the proposals was about the quality of services.
The solution proposed is to extend the information of the services with 
information about the status of the service (Ok, no tested, no 
responding, etc)

The services without "enough" quality will be hidden from the general 
list giving the user always services that work properly. The hidden one 
will still available if the user ask for them so they can be 
deregistered or update their information for make them back to work.

I'm preparing a more detailed document that I will sent soon.
Best regards
Sergio.

Pierre-Yves Chibon wrote:
> Sebastien Carrere wrote:
>> Here is my list of Dead services (Ihave got also list for the 3 past 
>> days for testRegistry and Inab registry) :
>> http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral 
>>
>
>
> Just by curiosity :
>  - do you have a list of working services ?
>  - do you have available your results from the testing repository ? 
> (by the way, why are you testing it ?)
>
> Thanks in advance,
>
> Regards,
>
> Pierre
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 


From edward.kawas at gmail.com  Thu Nov  6 13:37:55 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Thu, 6 Nov 2008 10:37:55 -0800
Subject: [MOBY-dev] XML parsing takes forever
In-Reply-To: <Pine.LNX.4.64.0811051418290.27319@midas.psb.ugent.be>
References: <Pine.LNX.4.64.0811041403160.27319@midas.psb.ugent.be>
	<Pine.LNX.4.64.0811051418290.27319@midas.psb.ugent.be>
Message-ID: <4913398f.02578c0a.5c4f.ffff8bc1@mx.google.com>

Hi Kevin,

I applied your patch to Utils.pm. I also modified
MOSES::MOBY::Cache::Registries so that a file read isn?t done every time
that module is instantiated. Finally, I moved IO::Scalar to IO::String.

Now, instead of 1 hour+ to parse the XML file, we have got it down to just
over a minute. I am hesitant to remove the as_string overload in Base.pm. We
use it for displaying our datatypes nicely (better than if Data::Dumper was
used). If we can find a way to produce the same information as as_string and
be more efficient, I would swap that in right away.

The changes are in the CVS @ moby-live/Perl/MOSES-MOBY .

Please keep hacking away! I really appreciate all of your help so far!

Thanks.

Eddie



-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
Sent: November-05-08 7:57 AM
To: Core developer announcements
Subject: Re: [MOBY-dev] XML parsing takes forever

Hi,

after some more tests, I've succesfully reduced the parsing time from over
1 hour 15 minutes to 17 seconds. Which is not ideal yet, but at least
workable :)

This is the profiling result for the original environment. The test xml file
is the original file I've sent here without the sequence.

[davebowman at hal2000 ]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_no_dna.xml
Trying to parse.. Done.
[davebowman at hal2000 ]$ dprofpp
Total Elapsed Time = 4348.764 Seconds
   User+System Time = 4344.374 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  97.3   4227. 4227.2 399336   0.0011 0.0011  IO::Scalar::print
  0.88   38.06 4357.4 189130   0.0000 0.0023  Dumpvalue::unwrap
  0.80   34.54 51.532 369048   0.0000 0.0000  Dumpvalue::stringify
  0.54   23.63 23.634 463568   0.0000 0.0000  File::Spec::Unix::canonpath
  0.33   14.23 4356.7 188364   0.0000 0.0023  Dumpvalue::DumpElem


First I simply removed the overload operator "" (as_string). I still get the
same results and processing time is reduced drastically.
I don't really know why this way of stringifying (complex) perl types is
used here. 
Is there a special reason for it? Can't we simply use Data::Dumper (which
prints to a $var by default)?

[davebowman at hal2000 ]$ dprofpp
Total Elapsed Time = 35.85114 Seconds
   User+System Time = 33.54114 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  64.5   21.64 21.646 463568   0.0000 0.0000  File::Spec::Unix::canonpath
  22.9   7.698 31.602 231569   0.0000 0.0000  File::Spec::Unix::catfile
  19.7   6.607  6.607   2144   0.0031 0.0031
MOSES::MOBY::Cache::Registries::init
  18.6   6.251 37.772   2143   0.0029 0.0176
MOSES::MOBY::Generators::Utils::find_file
  6.80   2.282 13.448 231569   0.0000 0.0000  File::Spec::Unix::catdir

A lot of calls are made to MOSES::MOBY::Generators::Utils::find_file,
which in it's turn does a lot of calls with File::Spec. So I simply cache
the results.
Time is reduced to 17 seconds:

Total Elapsed Time = 17.34032 Seconds
   User+System Time = 13.20032 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  51.5   6.807  6.807   2144   0.0032 0.0032
MOSES::MOBY::Cache::Registries::init
  9.01   1.190  2.005 162858   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
  4.65   0.614  0.614 310751   0.0000 0.0000  XML::LibXML::Node::nodeName
  4.64   0.612  1.655  32171   0.0000 0.0001
XML::LibXML::Element::getChildrenByTagName
  4.48   0.592  0.592   5515   0.0001 0.0001
MOSES::MOBY::Cache::Central::_clean

More then half of the time still goes to building up the MOBY environment
for each processed element in the XML file. Isn't this a bit overkill? ;) Is
there a way to make a MOSES::MOBY factory that can handle the building up of
the moby environment just once in the same (mod_perl) call? Or is there a
reason this has been avoided?

I've included the patches for the result caching in
MOSES::MOBY::Generators::Utils::find_file . there's not much to it. My
version of MOSES::MOBY still is the one uploaded to cpan: Module MOSES::MOBY
(E/EK/EKAWAS/MOSES-MOBY-0.86.tar.gz)

happy hacking,
Kenny


On Tue, 4 Nov 2008, Kenny Billiau wrote:

> Hi,
>
> I've run a little testprogram to see where most of the time is spent. 
> I use a stripped down version of the xml file I've sent you. All but a 
> few ElementContainers and no sequence. Just to keep the processing 
> time low enough to debug.
>
>     use MOSES::MOBY::Parser;
>
>       open(FILE, '<', $ARGV[0]);
>       my @lines = <FILE>;
>       close FILE;
>
>       my $parser = MOSES::MOBY::Parser->new();
>       print $parser->parse( method => 'string', data => join(q{}, 
> @lines) );
>
>
> [davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl 
> mobysav_small.xml [davebowman at hal2000 ~]$ dprofpp 
> Exporter::Heavy::heavy_export has 14 unstacked calls in outer 
> Config::Simple::import_from has 1 unstacked calls in outer 
> utf8::AUTOLOAD has -1 unstacked calls in outer
> Config::Simple::  verbose has 1 unstacked calls in outer
> Config::Simple::  vars has 1 unstacked calls in outer 
> AutoLoader::AUTOLOAD has -3 unstacked calls in outer
> Fcntl::  AUTOLOAD has -6 unstacked calls in outer
> Fcntl: :__ANON__ has 6 unstacked calls in outer Exporter::export has 
> -14 unstacked calls in outer utf8::SWASHNEW has 1 unstacked calls in 
> outer Total Elapsed Time = 8.470220 Seconds  User+System Time = 
> 7.750220 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call 
> Csec/c  Name
>   68.1   5.280  5.280  69736   0.0001 0.0001  IO::Scalar::print
>   6.94   0.538  0.783  64088   0.0000 0.0000  Dumpvalue::stringify
>   6.04   0.468  7.173  33755   0.0000 0.0002  Dumpvalue::unwrap
>   4.30   0.333  0.333  47091   0.0000 0.0000  File::Spec::Unix::canonpath
>   2.43   0.188  7.157  33006   0.0000 0.0002  Dumpvalue::DumpElem
>   2.13   0.165  0.165  59776   0.0000 0.0000  overload::mycan
>   1.78   0.138  0.224  15990   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
>   1.42   0.110  0.708    539   0.0002 0.0013
>   MOSES::MOBY::Cache::Central::_createDataTypeFromXML
>   1.39   0.108  0.108 142023   0.0000 0.0000  UNIVERSAL::isa
>   1.29   0.100  0.100    208   0.0005 0.0005
>   MOSES::MOBY::Cache::Registries::init
>   1.20   0.093  0.357  17359   0.0000 0.0000
overload::OverloadedStringify
>   1.16   0.090  0.090  30407   0.0000 0.0000  XML::LibXML::Node::nodeName
>   1.02   0.079  0.079    539   0.0001 0.0001
>   MOSES::MOBY::Cache::Central::_clean
>   0.99   0.077  0.418    207   0.0004 0.0020
>   MOSES::MOBY::Generators::Utils::find_file
>   0.94   0.073  0.203   3147   0.0000 0.0001
>   XML::LibXML::Element::getChildrenByTagName
>
> It's fairly obivous now ;)
>
> If I remove the overload operator in MOSES::MOBY::Base, time is 
> reduced 15fold! But for the 250KB file I've sent, it still takes us a 
> minute to parse.
>
> If I simply parse the file with XML::LibXML and no callbacks or custom 
> handlers, it's parsed instantly:
>
> [davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl 
> mobysav_small.xml
> [kebil at psbdev01 ~]$ dprofpp
> Total Elapsed Time = 0.079034 Seconds
>    User+System Time = 0.079034 Seconds
>
>
> -Kenny, slowly getting to the bottom of this ;)
>
>
> On Thu, 30 Oct 2008, Edward Kawas wrote:
>
>>   Now I can confirm that it is slow parsing and not an infinite loop! The
>>   XML
>>   file took over an hour on my virtual machine to parse. I will see what
I
>>   can
>>   do with other SAX parsers for Perl.
>>
>>   Thanks,
>>
>>   Eddie
>>
>>   -----Original Message-----
>>   From: moby-dev-bounces at lists.open-bio.org
>>   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Jos? Mar?a
>>   Fern?ndez Gonz?lez
>>   Sent: October-30-08 12:55 PM
>>   To: Core developer announcements
>>   Subject: Re: [MOBY-dev] XML parsing takes forever
>>
>>   Hi everybody,
>>   	I'm using the XML::LibXML SAX parser for other projects and tasks,
>>   and it can
>>   be really, really slow. You don't realize that until you are parsing a
>>   large
>>   file from a pipe, and you see that most of the time the program which
is
>>   parsing is CPU (and not I/O) bounded.
>>
>>    Best Regards,
>>     Jos? Mar?a
>>
>>   Edward Kawas wrote:
>> >   I think that you are right about the parser not going into an 
>> > infinite
>>   loop.
>> >   For some reason, the SAX parser seems to be really slow (although, I 
>> >   don't
>> >   want to say that until I know for sure).
>> > 
>> >   Eddie
>> > 
>> >   -----Original Message-----
>> >   From: moby-dev-bounces at lists.open-bio.org
>> >   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny
Billiau
>> >   Sent: October-30-08 11:56 AM
>> >   To: Core developer announcements
>> >   Subject: Re: [MOBY-dev] XML parsing takes forever
>> > 
>> >   Hi,
>> > 
>> >   thx for helping to investigate the problem!
>> > 
>> >   I'm pretty sure the parser gets out of it's mess. When you truncate
the
>> >   XML file by removing all but a few ElementContainers, it takes
awhile, 
>> >   but
>> 
>> >   it actually returns without error.
>> > 
>> >   And memory consuption doesn't seem to be a problem either, as mostly 
>> >   'top'
>> 
>> >   reveals that the invoked webservice script uses a steady amount of 
>> >   memory.
>> > 
>> >   thx,
>> >   Kenny
>> > 
>> >   On Thu, 30 Oct 2008, Edward Kawas wrote:
>> > 
>> > >   So this one might take some time... from what I can see, our parser

>> > >   goes
>> > >   into an infinite loop (although, I am not certain about that ...).
>> > > 
>> > >   I will play around with this XML file and get back to you.
>> > > 
>> > >   Thanks,
>> > > 
>> > >   Eddie
>> > > 
>> > >   -----Original Message-----
>> > >   From: moby-dev-bounces at lists.open-bio.org
>> > >   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny 
>> > >   Billiau
>> > >   Sent: October-30-08 8:03 AM
>> > >   To: Core developer announcements
>> > >   Subject: Re: [MOBY-dev] XML parsing takes forever
>> > > 
>> > >   Hi,
>> > > 
>> > >   here you go,
>> > > 
>> > >   Kenny
>> > > 
>> > >   On Thu, 30 Oct 2008, Edward Kawas wrote:
>> > > 
>> > > >   Hi,
>> > > > 
>> > > >   This is the right list! Could you send me the XML for your 
>> > > > second
>> >   service?
>> > > >   Thanks,
>> > > > 
>> > > >   Eddie
>> > >  _______________________________________________
>> > >   MOBY-dev mailing list
>> > >   MOBY-dev at lists.open-bio.org
>> > >   http://lists.open-bio.org/mailman/listinfo/moby-dev
>> > > 
>> >
>> 
>> 
>
>

-- 



From groscurt at mpiz-koeln.mpg.de  Fri Nov  7 03:56:53 2008
From: groscurt at mpiz-koeln.mpg.de (groscurt)
Date: Fri, 7 Nov 2008 09:56:53 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49131B32.2010605@pingoured.fr>
Message-ID: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>

Mhm i missed the intial email to that respond.

so i'm wondering - what is different from your list to the list from the one
from mobycentral http://moby.ucalgary.ca/moby/ValidateService ? 

And to the discussion about "working proper" services - it might be in the
document you will see, Sergio, but i'm really curios about the definition of
the phrase "work properly". So when does a service work properly ? 

of course, in my point of view, if it fulfills its registrated defintion -
so the question is how to test that ? 

I guess i wait for the document sergio was talking about ;-)

Cheers
andreas

----------------original message-----------------
From: "Pierre-Yves Chibon" 
To: "Core developer announcements" moby-dev at lists.open-bio.org
Date: Thu, 06 Nov 2008 17:28:34 +0100
-------------------------------------------------
> Sebastien Carrere wrote:
>> Here is my list of Dead services (Ihave got also list for the 3 past 
>> days for testRegistry and Inab registry) :
>>
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral
> 
> 
> Just by curiosity :
>   - do you have a list of working services ?
>   - do you have available your results from the testing repository ? (by 
> the way, why are you testing it ?)
> 
> Thanks in advance,
> 
> Regards,
> 
> Pierre
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
> 




From Sebastien.Carrere at toulouse.inra.fr  Fri Nov  7 04:28:32 2008
From: Sebastien.Carrere at toulouse.inra.fr (Sebastien Carrere)
Date: Fri, 07 Nov 2008 10:28:32 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
Message-ID: <49140A40.5070102@toulouse.inra.fr>

Hi,

my list contains the DeadServices provided on biomoby.org but also 
services where the following Perl test fails:

eval {$service_instance->execute(XMLinputlist =>[[]])};

The most frequent error is "Service execution failed: 500 Can't connect 
to localhost:8080 (connect: Connection refused)"

I don't know what is exactly done by 
http://moby.ucalgary.ca/moby/ValidateService .

Nevertheless, if we want to test if a webservice "works properly", we 
should first ask service developpers to register with test data.
This is already what we do using the PlayMoby framework.
By this way we survey our services ('not tested', 'ok', 'down') and 
generate a daily report.


Sebastien




groscurt wrote:
> Mhm i missed the intial email to that respond.
>
> so i'm wondering - what is different from your list to the list from the one
> from mobycentral http://moby.ucalgary.ca/moby/ValidateService ? 
>
> And to the discussion about "working proper" services - it might be in the
> document you will see, Sergio, but i'm really curios about the definition of
> the phrase "work properly". So when does a service work properly ? 
>
> of course, in my point of view, if it fulfills its registrated defintion -
> so the question is how to test that ? 
>
> I guess i wait for the document sergio was talking about ;-)
>
> Cheers
> andreas
>
> ----------------original message-----------------
> From: "Pierre-Yves Chibon" 
> To: "Core developer announcements" moby-dev at lists.open-bio.org
> Date: Thu, 06 Nov 2008 17:28:34 +0100
> -------------------------------------------------
>   
>> Sebastien Carrere wrote:
>>     
>>> Here is my list of Dead services (Ihave got also list for the 3 past 
>>> days for testRegistry and Inab registry) :
>>>
>>>       
> http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral
>   
>> Just by curiosity :
>>   - do you have a list of working services ?
>>   - do you have available your results from the testing repository ? (by 
>> the way, why are you testing it ?)
>>
>> Thanks in advance,
>>
>> Regards,
>>
>> Pierre
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>>     
>
>
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>   

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Sebastien_Carrere.vcf
Type: text/x-vcard
Size: 387 bytes
Desc: not available
URL: <http://lists.open-bio.org/pipermail/moby-dev/attachments/20081107/9e7a529d/attachment.vcf>

From groscurt at mpiz-koeln.mpg.de  Fri Nov  7 05:20:11 2008
From: groscurt at mpiz-koeln.mpg.de (groscurt)
Date: Fri, 7 Nov 2008 11:20:11 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49140A40.5070102@toulouse.inra.fr>
Message-ID: <6371f6e1ab64fc3fdd519aeb0378eaa7@mpizmail.mpiz-koeln.mpg.de>

> Nevertheless, if we want to test if a webservice "works properly", we 
> should first ask service developpers to register with test data.
if i remember correctly, this was something we discussed in tokyo. 
I think we should consider to force people to give an example input for the
service, as this is nice a) for testing and b) to understand what the
service does.

But i remember one person (sorry I'm horrible with names) gave a good point
about service which need huge compute power and run for a longer time, which
then would bring them to their knees only because someone wants to test the
"integrity" of the service.

So service providers should be able to say whether those testing shall be
done for their service(s) or not, knowing that their services might be
highlighted as "not tested", but no "not working properly".

andreas

PS: It would be even nicer to force people more. E.g. if someone offers a
blast or any other algorithm to provide the information about what version
of the algorithm the service use or which database (in case of blast for
example) etc... but this can be become complicated.




From srramirez at uma.es  Fri Nov  7 09:11:30 2008
From: srramirez at uma.es (Sergio Ramirez Ramirez)
Date: Fri, 07 Nov 2008 15:11:30 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49140A40.5070102@toulouse.inra.fr>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr>
Message-ID: <49144C92.9020903@uma.es>

Hello all,

The only difference of the proposal with the list 
"http://moby.ucalgary.ca/moby/ValidateService"  is to include the 
information of this list in the service repository so it can be accessed 
using the API of MOBY.
It also provides the advantage of filter those services that haven't 
been validate when the user ask for a list of services.

When the service hasn't and example we only can test is the service is 
answering or not, so that is what means "work properly". In the INB we 
have examples associated with the services so is possible to test the 
services with more detail.

For non-sample services the status could be: Online, Timeout is the 
service is not answering and BadName when the service name is wrong. 
When an example was available is possible to check if the example is 
correct and the service doesn't returns with errors;  if the output type 
is the expected and/or if the output itself is correct using some 
mechanism to compare it with the one registered as example.

This is the new proposal in big details, but the main idea is to have 
the information that is now in the list available directly for clients.

Best regards,
Sergio.


Sebastien Carrere wrote:
> Hi,
>
> my list contains the DeadServices provided on biomoby.org but also 
> services where the following Perl test fails:
>
> eval {$service_instance->execute(XMLinputlist =>[[]])};
>
> The most frequent error is "Service execution failed: 500 Can't 
> connect to localhost:8080 (connect: Connection refused)"
>
> I don't know what is exactly done by 
> http://moby.ucalgary.ca/moby/ValidateService .
>
> Nevertheless, if we want to test if a webservice "works properly", we 
> should first ask service developpers to register with test data.
> This is already what we do using the PlayMoby framework.
> By this way we survey our services ('not tested', 'ok', 'down') and 
> generate a daily report.
>
>
> Sebastien
>
>
>
>
> groscurt wrote:
>> Mhm i missed the intial email to that respond.
>>
>> so i'm wondering - what is different from your list to the list from 
>> the one
>> from mobycentral http://moby.ucalgary.ca/moby/ValidateService ?
>> And to the discussion about "working proper" services - it might be 
>> in the
>> document you will see, Sergio, but i'm really curios about the 
>> definition of
>> the phrase "work properly". So when does a service work properly ?
>> of course, in my point of view, if it fulfills its registrated 
>> defintion -
>> so the question is how to test that ?
>> I guess i wait for the document sergio was talking about ;-)
>>
>> Cheers
>> andreas
>>
>> ----------------original message-----------------
>> From: "Pierre-Yves Chibon" To: "Core developer announcements" 
>> moby-dev at lists.open-bio.org
>> Date: Thu, 06 Nov 2008 17:28:34 +0100
>> -------------------------------------------------
>>  
>>> Sebastien Carrere wrote:
>>>    
>>>> Here is my list of Dead services (Ihave got also list for the 3 
>>>> past days for testRegistry and Inab registry) :
>>>>
>>>>       
>> http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral 
>>
>>  
>>> Just by curiosity :
>>>   - do you have a list of working services ?
>>>   - do you have available your results from the testing repository ? 
>>> (by the way, why are you testing it ?)
>>>
>>> Thanks in advance,
>>>
>>> Regards,
>>>
>>> Pierre
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>
>>>     
>>
>>
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>   
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>   


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 


From edward.kawas at gmail.com  Fri Nov  7 09:39:42 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Fri, 7 Nov 2008 06:39:42 -0800
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49144C92.9020903@uma.es>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>	<49140A40.5070102@toulouse.inra.fr>
	<49144C92.9020903@uma.es>
Message-ID: <4914533d.1f538c0a.541e.5585@mx.google.com>

Hello,

I wanted to chime in with my 2 cents!

Back in February during the Biohackathon, we (Canadian/Spanish/German
groups) came up with some predicates that we could use to add unit testing
information to a service[1]. 

Currently, JAVA can support parsing the RDF to get this information as well
as comparing the unit test information with the output for the service. I
have yet to add Perl support.

In addition, I have been working on a Dashboard panel that will help users
fill in unit test information for a particular service. In addition to
filling in the information, a service user can test the selected service and
save the unit test (in the service RDF).

Is the current proposal for testing services based on what we came up with
in February?

Thanks,

Eddie

[1] http://lists.open-bio.org/pipermail/moby-dev/2008-June/005061.html


-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Sergio Ramirez
Ramirez
Sent: November-07-08 6:12 AM
To: Core developer announcements
Subject: Re: [MOBY-dev] Dead services and how to figure them out

Hello all,

The only difference of the proposal with the list 
"http://moby.ucalgary.ca/moby/ValidateService"  is to include the 
information of this list in the service repository so it can be accessed 
using the API of MOBY.
It also provides the advantage of filter those services that haven't 
been validate when the user ask for a list of services.

When the service hasn't and example we only can test is the service is 
answering or not, so that is what means "work properly". In the INB we 
have examples associated with the services so is possible to test the 
services with more detail.

For non-sample services the status could be: Online, Timeout is the 
service is not answering and BadName when the service name is wrong. 
When an example was available is possible to check if the example is 
correct and the service doesn't returns with errors;  if the output type 
is the expected and/or if the output itself is correct using some 
mechanism to compare it with the one registered as example.

This is the new proposal in big details, but the main idea is to have 
the information that is now in the list available directly for clients.

Best regards,
Sergio.


Sebastien Carrere wrote:
> Hi,
>
> my list contains the DeadServices provided on biomoby.org but also 
> services where the following Perl test fails:
>
> eval {$service_instance->execute(XMLinputlist =>[[]])};
>
> The most frequent error is "Service execution failed: 500 Can't 
> connect to localhost:8080 (connect: Connection refused)"
>
> I don't know what is exactly done by 
> http://moby.ucalgary.ca/moby/ValidateService .
>
> Nevertheless, if we want to test if a webservice "works properly", we 
> should first ask service developpers to register with test data.
> This is already what we do using the PlayMoby framework.
> By this way we survey our services ('not tested', 'ok', 'down') and 
> generate a daily report.
>
>
> Sebastien
>
>
>
>
> groscurt wrote:
>> Mhm i missed the intial email to that respond.
>>
>> so i'm wondering - what is different from your list to the list from 
>> the one
>> from mobycentral http://moby.ucalgary.ca/moby/ValidateService ?
>> And to the discussion about "working proper" services - it might be 
>> in the
>> document you will see, Sergio, but i'm really curios about the 
>> definition of
>> the phrase "work properly". So when does a service work properly ?
>> of course, in my point of view, if it fulfills its registrated 
>> defintion -
>> so the question is how to test that ?
>> I guess i wait for the document sergio was talking about ;-)
>>
>> Cheers
>> andreas
>>
>> ----------------original message-----------------
>> From: "Pierre-Yves Chibon" To: "Core developer announcements" 
>> moby-dev at lists.open-bio.org
>> Date: Thu, 06 Nov 2008 17:28:34 +0100
>> -------------------------------------------------
>>  
>>> Sebastien Carrere wrote:
>>>    
>>>> Here is my list of Dead services (Ihave got also list for the 3 
>>>> past days for testRegistry and Inab registry) :
>>>>
>>>>       
>>
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral 
>>
>>  
>>> Just by curiosity :
>>>   - do you have a list of working services ?
>>>   - do you have available your results from the testing repository ? 
>>> (by the way, why are you testing it ?)
>>>
>>> Thanks in advance,
>>>
>>> Regards,
>>>
>>> Pierre
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>
>>>     
>>
>>
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>   
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>   


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 

_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev



From srramirez at uma.es  Mon Nov 10 11:54:32 2008
From: srramirez at uma.es (Sergio Ramirez Ramirez)
Date: Mon, 10 Nov 2008 17:54:32 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <4914533d.1f538c0a.541e.5585@mx.google.com>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>	<49140A40.5070102@toulouse.inra.fr>	<49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com>
Message-ID: <49186748.5090402@uma.es>

Hi Eddie,

The idea is to define a list of status for the services, direct 
accessible from the list of them, that represents the results of the 
tests make over it.
If there aren't samples available, the only test that can be done is a 
ping to see is the service responds. In this case the status proposed are:

    * online: The service responds properly to the ping 
    * Timeout: If the time limit is got when trying to connect to the
      service.
    * NotFound: The service url is not found in the server
    * InternalServiceError: The service responds with an error related
      with the implementation of the service.
    * Disconnected: The connection with the service couldn't be established.

In addition, the NonAvailable status represents those services that have 
been down manually by the service provider for performing administrative 
tasks; and the unknown status when cannot be determine the status of the 
service.

When an example was available (defined in the RDF or in the extended 
registry of the INB, etc) can be make more specific test to the service. 
In this case, the set that we proposed is:

    * Tested: The service has been tested with the inputs given and no
      errors have been reported.
    * BadSample: The service respond with errors related with the input
      given.
    * NoSample: There aren't example for testing the service.

The output can be also used for test the behavior of the service. In 
this case we discussed two ways for testing:

    * TypeCorrect: The type of the data returned is the same or
      compatible with the expected one.
    * TypeIncorrect: If the data type is not compatible with the
      expected one.

And if  a way is provided for check the output content (regular 
expression or xpath) them there are two more status to contemplate:

    * OutputTested: The output content is check and fit with the results
      expected.
    * OutputFailed: in other case.

As you can see Eddie, the proposal is compatible with the one you 
mention. The only difference is the way to query the status of the 
service, that know can  be accessible using the functions of the MOBY 
Central..

Edward Kawas wrote:
> Hello,
>
> I wanted to chime in with my 2 cents!
>
> Back in February during the Biohackathon, we (Canadian/Spanish/German
> groups) came up with some predicates that we could use to add unit testing
> information to a service[1]. 
>
> Currently, JAVA can support parsing the RDF to get this information as well
> as comparing the unit test information with the output for the service. I
> have yet to add Perl support.
>
> In addition, I have been working on a Dashboard panel that will help users
> fill in unit test information for a particular service. In addition to
> filling in the information, a service user can test the selected service and
> save the unit test (in the service RDF).
>
> Is the current proposal for testing services based on what we came up with
> in February?
>
> Thanks,
>
> Eddie
>
> [1] http://lists.open-bio.org/pipermail/moby-dev/2008-June/005061.html
>
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Sergio Ramirez
> Ramirez
> Sent: November-07-08 6:12 AM
> To: Core developer announcements
> Subject: Re: [MOBY-dev] Dead services and how to figure them out
>
> Hello all,
>
> The only difference of the proposal with the list 
> "http://moby.ucalgary.ca/moby/ValidateService"  is to include the 
> information of this list in the service repository so it can be accessed 
> using the API of MOBY.
> It also provides the advantage of filter those services that haven't 
> been validate when the user ask for a list of services.
>
> When the service hasn't and example we only can test is the service is 
> answering or not, so that is what means "work properly". In the INB we 
> have examples associated with the services so is possible to test the 
> services with more detail.
>
> For non-sample services the status could be: Online, Timeout is the 
> service is not answering and BadName when the service name is wrong. 
> When an example was available is possible to check if the example is 
> correct and the service doesn't returns with errors;  if the output type 
> is the expected and/or if the output itself is correct using some 
> mechanism to compare it with the one registered as example.
>
> This is the new proposal in big details, but the main idea is to have 
> the information that is now in the list available directly for clients.
>
> Best regards,
> Sergio.
>
>
> Sebastien Carrere wrote:
>   
>> Hi,
>>
>> my list contains the DeadServices provided on biomoby.org but also 
>> services where the following Perl test fails:
>>
>> eval {$service_instance->execute(XMLinputlist =>[[]])};
>>
>> The most frequent error is "Service execution failed: 500 Can't 
>> connect to localhost:8080 (connect: Connection refused)"
>>
>> I don't know what is exactly done by 
>> http://moby.ucalgary.ca/moby/ValidateService .
>>
>> Nevertheless, if we want to test if a webservice "works properly", we 
>> should first ask service developpers to register with test data.
>> This is already what we do using the PlayMoby framework.
>> By this way we survey our services ('not tested', 'ok', 'down') and 
>> generate a daily report.
>>
>>
>> Sebastien
>>
>>
>>
>>
>> groscurt wrote:
>>     
>>> Mhm i missed the intial email to that respond.
>>>
>>> so i'm wondering - what is different from your list to the list from 
>>> the one
>>> from mobycentral http://moby.ucalgary.ca/moby/ValidateService ?
>>> And to the discussion about "working proper" services - it might be 
>>> in the
>>> document you will see, Sergio, but i'm really curios about the 
>>> definition of
>>> the phrase "work properly". So when does a service work properly ?
>>> of course, in my point of view, if it fulfills its registrated 
>>> defintion -
>>> so the question is how to test that ?
>>> I guess i wait for the document sergio was talking about ;-)
>>>
>>> Cheers
>>> andreas
>>>
>>> ----------------original message-----------------
>>> From: "Pierre-Yves Chibon" To: "Core developer announcements" 
>>> moby-dev at lists.open-bio.org
>>> Date: Thu, 06 Nov 2008 17:28:34 +0100
>>> -------------------------------------------------
>>>  
>>>       
>>>> Sebastien Carrere wrote:
>>>>    
>>>>         
>>>>> Here is my list of Dead services (Ihave got also list for the 3 
>>>>> past days for testRegistry and Inab registry) :
>>>>>
>>>>>       
>>>>>           
> http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral 
>   
>>>  
>>>       
>>>> Just by curiosity :
>>>>   - do you have a list of working services ?
>>>>   - do you have available your results from the testing repository ? 
>>>> (by the way, why are you testing it ?)
>>>>
>>>> Thanks in advance,
>>>>
>>>> Regards,
>>>>
>>>> Pierre
>>>> _______________________________________________
>>>> MOBY-dev mailing list
>>>> MOBY-dev at lists.open-bio.org
>>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>>
>>>>     
>>>>         
>>>
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>   
>>>       
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>   
>>     
>
>
>   


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 


From markw at illuminae.com  Mon Nov 10 12:50:24 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Mon, 10 Nov 2008 09:50:24 -0800
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49186748.5090402@uma.es>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr> <49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com> <49186748.5090402@uma.es>
Message-ID: <op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>

On Mon, 10 Nov 2008 08:54:32 -0800, Sergio Ramirez Ramirez  
<srramirez at uma.es> wrote:


>   * Timeout: If the time limit is got when trying to connect to the
>       service.
>     * Disconnected: The connection with the service couldn't be  
> established.

What's the difference between these two?



> As you can see Eddie, the proposal is compatible with the one you  
> mention. The only difference is the way to query the status of the  
> service, that know can  be accessible using the functions of the MOBY  
> Central..


Thanks Sergio!  As usual, the INB has put together a very clear and  
thought-out proposal, and it is perfectly consistent with what we had in  
mind!  fantastic!

If I understand correctly, you currently keep this kind of metadata in the  
INB extended MOBY Central, right?  How tightly integrated is it with MOBY  
Central?  (i.e. have you changed the MOBY Central API such that you query  
for this metadata inside of the functions defined by the 'official' API,  
or is this data accessible via new API functions?)

We (my lab) are similarly interested in QoS issues, and where to store  
this kind of QoSD metadata.  We're looking at repositories such as  
BioCatalogue as a possibile store, since we don't think that this is the  
kind of metadata that the registry itself should be in control of (in  
fact, the registry may not even be capable of knowing!)

Do you or your colleagues at INB have strong opinions about where to store  
this metadata, and what the API should be to query for it?

best wishes!

Mark


From srramirez at uma.es  Mon Nov 10 13:37:25 2008
From: srramirez at uma.es (Sergio Ramirez Ramirez)
Date: Mon, 10 Nov 2008 19:37:25 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>	<49140A40.5070102@toulouse.inra.fr>
	<49144C92.9020903@uma.es>	<4914533d.1f538c0a.541e.5585@mx.google.com>
	<49186748.5090402@uma.es>
	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
Message-ID: <49187F65.7040502@uma.es>

Thanks Mark for your answer.

I respond "between lines":

Mark Wilkinson wrote:
> On Mon, 10 Nov 2008 08:54:32 -0800, Sergio Ramirez Ramirez 
> <srramirez at uma.es> wrote:
>
>
>>   * Timeout: If the time limit is got when trying to connect to the
>>       service.
>>     * Disconnected: The connection with the service couldn't be 
>> established.
>
> What's the difference between these two?
>
Really disconnected is more general than Timeout. It can be used when is 
not possible to connect with the service but not exactly due a timeout.
>
>
>> As you can see Eddie, the proposal is compatible with the one you 
>> mention. The only difference is the way to query the status of the 
>> service, that know can  be accessible using the functions of the MOBY 
>> Central..
>
>
> Thanks Sergio!  As usual, the INB has put together a very clear and 
> thought-out proposal, and it is perfectly consistent with what we had 
> in mind!  fantastic!
>
> If I understand correctly, you currently keep this kind of metadata in 
> the INB extended MOBY Central, right?  How tightly integrated is it 
> with MOBY Central?  (i.e. have you changed the MOBY Central API such 
> that you query for this metadata inside of the functions defined by 
> the 'official' API, or is this data accessible via new API functions?)
The idea is to make the information of the test done accessible thought 
the API. For that  we need to include the status and the last test done 
as new information of the registry. Related with the functions, the 
proposal was to extend the functions to recover the list of services so 
they returns also information about the status. Also was proposed that 
those services that don't pass the test weren't returned in the list by 
default so the users always got a list of working services
>
> We (my lab) are similarly interested in QoS issues, and where to store 
> this kind of QoSD metadata.  We're looking at repositories such as 
> BioCatalogue as a possibile store, since we don't think that this is 
> the kind of metadata that the registry itself should be in control of 
> (in fact, the registry may not even be capable of knowing!)
> Do you or your colleagues at INB have strong opinions about where to 
> store this metadata, and what the API should be to query for it?
In my personal opinion the information have to be stored in accesible 
way (web-service, xml or similar) so can be handle programmability. The 
real format and protocol used for stored them is not the most important 
thing.
>
> best wishes!
>
> Mark
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 


From markw at illuminae.com  Mon Nov 10 14:02:59 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Mon, 10 Nov 2008 11:02:59 -0800
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49187F65.7040502@uma.es>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr> <49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com> <49186748.5090402@uma.es>
	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
	<49187F65.7040502@uma.es>
Message-ID: <op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>

Hi Sergio!

it looks like we're back to the recurring question:  what metadata belongs  
IN the registry, and what metadata belongs elsewhere...

     <<sigh>>

I tend to have the opinion that only metadata necessary for discovery of a  
service should be in the registry itself, and that metadata about QoS,  
uptime, downtime, versioning, etc. all belongs somewhere else...  by  
"somewhere else" I mean (a) in the hands of the service provider  
themselves, or (b) in a third-party metadata repository.  This was one of  
the primary driving motivations behind our use of LSIDs in BioMoby, since  
the LSID allowed us to provide this non-registry metadata through a  
straightforward, predictable API that was independent of the registry  
API.  That way, we didn't have to modify the registry or the registry API  
every time we had a new kind of metadata!  I *still* believe this is the  
right choice... though I understand that sometimes efficiency of query is  
a compelling reason to have all of that metadata in the same place.   
Nevertheless, I don't think it should be a function of the registry to  
decide which services it will and will not show you!  The registry should  
simply tell you everything it knows, and it should be a client-side  
decision which of those services it displays to the end-user (IMO).  As  
such, I don't necessarily see the motivation for having this kind of  
metadata as part of the registry API...

How do others feel about this?  Is there *anyone* (other than Gbrowse  
Moby) who actually uses the LSID resolver at MOBY Central to retrieve this  
third-party metadata?  If not, is there a reason why not?

Martin, perhaps you can weigh-in on this question, as you had  
once-upon-a-time worked on the NetNanny project that would have monitored  
QoS... do you have a strong opinion one way or the other about where this  
metadata should live?  (LOL!  Asking Martin if he has a "strong opinion"  
is perhaps redundant ;-)  LOL!)

Best wishes all!

Mark








On Mon, 10 Nov 2008 10:37:25 -0800, Sergio Ramirez Ramirez  
<srramirez at uma.es> wrote:

> Thanks Mark for your answer.
>
> I respond "between lines":
>
> Mark Wilkinson wrote:
>> On Mon, 10 Nov 2008 08:54:32 -0800, Sergio Ramirez Ramirez  
>> <srramirez at uma.es> wrote:
>>
>>
>>>   * Timeout: If the time limit is got when trying to connect to the
>>>       service.
>>>     * Disconnected: The connection with the service couldn't be  
>>> established.
>>
>> What's the difference between these two?
>>
> Really disconnected is more general than Timeout. It can be used when is  
> not possible to connect with the service but not exactly due a timeout.
>>
>>
>>> As you can see Eddie, the proposal is compatible with the one you  
>>> mention. The only difference is the way to query the status of the  
>>> service, that know can  be accessible using the functions of the MOBY  
>>> Central..
>>
>>
>> Thanks Sergio!  As usual, the INB has put together a very clear and  
>> thought-out proposal, and it is perfectly consistent with what we had  
>> in mind!  fantastic!
>>
>> If I understand correctly, you currently keep this kind of metadata in  
>> the INB extended MOBY Central, right?  How tightly integrated is it  
>> with MOBY Central?  (i.e. have you changed the MOBY Central API such  
>> that you query for this metadata inside of the functions defined by the  
>> 'official' API, or is this data accessible via new API functions?)
> The idea is to make the information of the test done accessible thought  
> the API. For that  we need to include the status and the last test done  
> as new information of the registry. Related with the functions, the  
> proposal was to extend the functions to recover the list of services so  
> they returns also information about the status. Also was proposed that  
> those services that don't pass the test weren't returned in the list by  
> default so the users always got a list of working services
>>
>> We (my lab) are similarly interested in QoS issues, and where to store  
>> this kind of QoSD metadata.  We're looking at repositories such as  
>> BioCatalogue as a possibile store, since we don't think that this is  
>> the kind of metadata that the registry itself should be in control of  
>> (in fact, the registry may not even be capable of knowing!)
>> Do you or your colleagues at INB have strong opinions about where to  
>> store this metadata, and what the API should be to query for it?
> In my personal opinion the information have to be stored in accesible  
> way (web-service, xml or similar) so can be handle programmability. The  
> real format and protocol used for stored them is not the most important  
> thing.
>>
>> best wishes!
>>
>> Mark
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>
>



-- 
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada

From martin.senger at gmail.com  Mon Nov 10 14:13:57 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 11 Nov 2008 00:43:57 +0530
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr> <49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com> <49186748.5090402@uma.es>
	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
	<49187F65.7040502@uma.es>
	<op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
Message-ID: <4d93f07c0811101113md33d8adif4c20ed4a6144f29@mail.gmail.com>

> I tend to have the opinion that only metadata necessary for discovery of a
> service should be in the registry itself, and that metadata about QoS,
> uptime, downtime, versioning, etc. all belongs somewhere else...  by
> "somewhere else" I mean (a) in the hands of the service provider themselves,
> or (b) in a third-party metadata repository.


Exactly my opinion (strong or weak :-)). Let's not pollute the registry. The
"smart" clients should anyway make some caches of the metadata so the
concern about the performance is a weak one I believe.

Cheers,
Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger

From jmfernandez at cnio.es  Mon Nov 10 14:33:09 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Mon, 10 Nov 2008 20:33:09 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <4d93f07c0811101113md33d8adif4c20ed4a6144f29@mail.gmail.com>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>	<49140A40.5070102@toulouse.inra.fr>
	<49144C92.9020903@uma.es>	<4914533d.1f538c0a.541e.5585@mx.google.com>
	<49186748.5090402@uma.es>	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>	<49187F65.7040502@uma.es>	<op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
	<4d93f07c0811101113md33d8adif4c20ed4a6144f29@mail.gmail.com>
Message-ID: <49188C75.1060707@cnio.es>

But, what does it happen if a service provider suddenly "disappears" (e.g. DNS
failure)? If QoS is stored just besides the service host, the Central Registry
could not fetch the QoS information in that case.

And, what does it happen if a QoS service provider suddenly "disappears" (e.g.
again DNS failure)? Service is usable, service is reachable, but QoS isn't...

Also, QoS is perceived in a different way depending on the place where your
client is plugged. The real QoS of a japanese service is different when you
are in Japan than when you are in Europe.

So, I have just realized that perhaps a possible answer is having replicated
QoS metadata servers spread around the world, related in some way to MOBY Central.

	Just my two euro cents...
		Jos? Mar?a

Martin Senger wrote:
>> I tend to have the opinion that only metadata necessary for discovery of a
>> service should be in the registry itself, and that metadata about QoS,
>> uptime, downtime, versioning, etc. all belongs somewhere else...  by
>> "somewhere else" I mean (a) in the hands of the service provider themselves,
>> or (b) in a third-party metadata repository.
> 
> 
> Exactly my opinion (strong or weak :-)). Let's not pollute the registry. The
> "smart" clients should anyway make some caches of the metadata so the
> concern about the performance is a weak one I believe.
> 
> Cheers,
> Martin
> 

-- 
"There is no reason why anybody would want a computer in their home" -
	Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???

"Premature optimization is the root of all evil." - Donald Knuth

Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es		Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n	Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029				Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3	Madrid (Spain)

**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.


From martin.senger at gmail.com  Mon Nov 10 21:48:00 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 11 Nov 2008 08:18:00 +0530
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49188C75.1060707@cnio.es>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr> <49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com> <49186748.5090402@uma.es>
	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
	<49187F65.7040502@uma.es>
	<op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
	<4d93f07c0811101113md33d8adif4c20ed4a6144f29@mail.gmail.com>
	<49188C75.1060707@cnio.es>
Message-ID: <4d93f07c0811101848v56734722j77c87c8a473f2550@mail.gmail.com>

> So, I have just realized that perhaps a possible answer is having
> replicated
> QoS metadata servers spread around the world


That's what I meant by "smart" clients caching it...

Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger

From schoof at mpiz-koeln.mpg.de  Tue Nov 11 03:05:42 2008
From: schoof at mpiz-koeln.mpg.de (Heiko Schoof)
Date: Tue, 11 Nov 2008 09:05:42 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr> <49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com>
	<49186748.5090402@uma.es>
	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
	<49187F65.7040502@uma.es>
	<op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
Message-ID: <8C3FFF0A-07AD-43FD-8BD0-E75619193E13@mpiz-koeln.mpg.de>

Hi Mark and others,

I tend to have strong opinions, but utter them softly (meaning both  
"hard to hear" and "gently" ;-)

I strongly lean towards keeping QoS separate from the core registry,  
many arguments have been given. But colocalization would make sense.  
We have been discussing failure redundancy for Moby Central, and a  
mirror plus QoS on the same server would make sense. If a client has  
no access to Central data, it doesn't need QoS. On the other hand, the  
central Central used for registration i.e. writing to the registry  
doesn't need QoS. That allows us to keep provider-provided metadata  
(like "this is the authoritative service for x" or "this is a mirror/ 
backup service for x") separate from real, externally measured QoS.

I actually might have a student that could put QoS awareness into our  
Jabba aggregator, which would make a lot of sense.

Greetings from Cologne in pouring rain,

HEiko

On 10.11.2008, at 20:02, Mark Wilkinson wrote:

> Hi Sergio!
>
> it looks like we're back to the recurring question:  what metadata  
> belongs IN the registry, and what metadata belongs elsewhere...
>
>    <<sigh>>
>
> I tend to have the opinion that only metadata necessary for  
> discovery of a service should be in the registry itself, and that  
> metadata about QoS, uptime, downtime, versioning, etc. all belongs  
> somewhere else...  by "somewhere else" I mean (a) in the hands of  
> the service provider themselves, or (b) in a third-party metadata  
> repository.  This was one of the primary driving motivations behind  
> our use of LSIDs in BioMoby, since the LSID allowed us to provide  
> this non-registry metadata through a straightforward, predictable  
> API that was independent of the registry API.  That way, we didn't  
> have to modify the registry or the registry API every time we had a  
> new kind of metadata!  I *still* believe this is the right choice...  
> though I understand that sometimes efficiency of query is a  
> compelling reason to have all of that metadata in the same place.   
> Nevertheless, I don't think it should be a function of the registry  
> to decide which services it will and will not show you!  The  
> registry should simply tell you everything it knows, and it should  
> be a client-side decision which of those services it displays to the  
> end-user (IMO).  As such, I don't necessarily see the motivation for  
> having this kind of metadata as part of the registry API...
>
> How do others feel about this?  Is there *anyone* (other than  
> Gbrowse Moby) who actually uses the LSID resolver at MOBY Central to  
> retrieve this third-party metadata?  If not, is there a reason why  
> not?
>
> Martin, perhaps you can weigh-in on this question, as you had once- 
> upon-a-time worked on the NetNanny project that would have monitored  
> QoS... do you have a strong opinion one way or the other about where  
> this metadata should live?  (LOL!  Asking Martin if he has a "strong  
> opinion" is perhaps redundant ;-)  LOL!)
>
> Best wishes all!
>
> Mark
>
>
>
>
>
>
>
>
> On Mon, 10 Nov 2008 10:37:25 -0800, Sergio Ramirez Ramirez <srramirez at uma.es 
> > wrote:
>
>> Thanks Mark for your answer.
>>
>> I respond "between lines":
>>
>> Mark Wilkinson wrote:
>>> On Mon, 10 Nov 2008 08:54:32 -0800, Sergio Ramirez Ramirez <srramirez at uma.es 
>>> > wrote:
>>>
>>>
>>>>  * Timeout: If the time limit is got when trying to connect to the
>>>>      service.
>>>>    * Disconnected: The connection with the service couldn't be  
>>>> established.
>>>
>>> What's the difference between these two?
>>>
>> Really disconnected is more general than Timeout. It can be used  
>> when is not possible to connect with the service but not exactly  
>> due a timeout.
>>>
>>>
>>>> As you can see Eddie, the proposal is compatible with the one you  
>>>> mention. The only difference is the way to query the status of  
>>>> the service, that know can  be accessible using the functions of  
>>>> the MOBY Central..
>>>
>>>
>>> Thanks Sergio!  As usual, the INB has put together a very clear  
>>> and thought-out proposal, and it is perfectly consistent with what  
>>> we had in mind!  fantastic!
>>>
>>> If I understand correctly, you currently keep this kind of  
>>> metadata in the INB extended MOBY Central, right?  How tightly  
>>> integrated is it with MOBY Central?  (i.e. have you changed the  
>>> MOBY Central API such that you query for this metadata inside of  
>>> the functions defined by the 'official' API, or is this data  
>>> accessible via new API functions?)
>> The idea is to make the information of the test done accessible  
>> thought the API. For that  we need to include the status and the  
>> last test done as new information of the registry. Related with the  
>> functions, the proposal was to extend the functions to recover the  
>> list of services so they returns also information about the status.  
>> Also was proposed that those services that don't pass the test  
>> weren't returned in the list by default so the users always got a  
>> list of working services
>>>
>>> We (my lab) are similarly interested in QoS issues, and where to  
>>> store this kind of QoSD metadata.  We're looking at repositories  
>>> such as BioCatalogue as a possibile store, since we don't think  
>>> that this is the kind of metadata that the registry itself should  
>>> be in control of (in fact, the registry may not even be capable of  
>>> knowing!)
>>> Do you or your colleagues at INB have strong opinions about where  
>>> to store this metadata, and what the API should be to query for it?
>> In my personal opinion the information have to be stored in  
>> accesible way (web-service, xml or similar) so can be handle  
>> programmability. The real format and protocol used for stored them  
>> is not the most important thing.
>>>
>>> best wishes!
>>>
>>> Mark
>>>
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>
>>
>>
>
>
>
> -- 
> Mark D Wilkinson, PI Bioinformatics
> Assistant Professor, Medical Genetics
> The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary  
> Research
> Providence Heart + Lung Institute
> University of British Columbia - St. Paul's Hospital
> Vancouver, BC, Canada
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev


From srramirez at uma.es  Tue Nov 11 04:53:53 2008
From: srramirez at uma.es (Sergio Ramirez Ramirez)
Date: Tue, 11 Nov 2008 10:53:53 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <8C3FFF0A-07AD-43FD-8BD0-E75619193E13@mpiz-koeln.mpg.de>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>	<49140A40.5070102@toulouse.inra.fr>
	<49144C92.9020903@uma.es>	<4914533d.1f538c0a.541e.5585@mx.google.com>	<49186748.5090402@uma.es>	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>	<49187F65.7040502@uma.es>	<op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
	<8C3FFF0A-07AD-43FD-8BD0-E75619193E13@mpiz-koeln.mpg.de>
Message-ID: <49195631.4090301@uma.es>

Hi Heiko, Mark and others

We can keep QoS separated from core registry as long as you can access 
automatically by clients, parsing RDF can be a possible solution.

Heiko Schoof wrote:
> Hi Mark and others,
>
> I tend to have strong opinions, but utter them softly (meaning both 
> "hard to hear" and "gently" ;-)
>
> I strongly lean towards keeping QoS separate from the core registry, 
> many arguments have been given. But colocalization would make sense. 
> We have been discussing failure redundancy for Moby Central, and a 
> mirror plus QoS on the same server would make sense. If a client has 
> no access to Central data, it doesn't need QoS. On the other hand, the 
> central Central used for registration i.e. writing to the registry 
> doesn't need QoS. That allows us to keep provider-provided metadata 
> (like "this is the authoritative service for x" or "this is a 
> mirror/backup service for x") separate from real, externally measured 
> QoS.
>
> I actually might have a student that could put QoS awareness into our 
> Jabba aggregator, which would make a lot of sense.
>
> Greetings from Cologne in pouring rain,
>
> HEiko
>
> On 10.11.2008, at 20:02, Mark Wilkinson wrote:
>
>> Hi Sergio!
>>
>> it looks like we're back to the recurring question:  what metadata 
>> belongs IN the registry, and what metadata belongs elsewhere...
>>
>>    <<sigh>>
>>
>> I tend to have the opinion that only metadata necessary for discovery 
>> of a service should be in the registry itself, and that metadata 
>> about QoS, uptime, downtime, versioning, etc. all belongs somewhere 
>> else...  by "somewhere else" I mean (a) in the hands of the service 
>> provider themselves, or (b) in a third-party metadata repository.  
>> This was one of the primary driving motivations behind our use of 
>> LSIDs in BioMoby, since the LSID allowed us to provide this 
>> non-registry metadata through a straightforward, predictable API that 
>> was independent of the registry API.  That way, we didn't have to 
>> modify the registry or the registry API every time we had a new kind 
>> of metadata!  I *still* believe this is the right choice... though I 
>> understand that sometimes efficiency of query is a compelling reason 
>> to have all of that metadata in the same place.  Nevertheless, I 
>> don't think it should be a function of the registry to decide which 
>> services it will and will not show you!  The registry should simply 
>> tell you everything it knows, and it should be a client-side decision 
>> which of those services it displays to the end-user (IMO).  As such, 
>> I don't necessarily see the motivation for having this kind of 
>> metadata as part of the registry API...
>>
>> How do others feel about this?  Is there *anyone* (other than Gbrowse 
>> Moby) who actually uses the LSID resolver at MOBY Central to retrieve 
>> this third-party metadata?  If not, is there a reason why not?
>>
>> Martin, perhaps you can weigh-in on this question, as you had 
>> once-upon-a-time worked on the NetNanny project that would have 
>> monitored QoS... do you have a strong opinion one way or the other 
>> about where this metadata should live?  (LOL!  Asking Martin if he 
>> has a "strong opinion" is perhaps redundant ;-)  LOL!)
>>
>> Best wishes all!
>>
>> Mark
>>
>>
>>
>>
>>
>>
>>
>>
>> On Mon, 10 Nov 2008 10:37:25 -0800, Sergio Ramirez Ramirez 
>> <srramirez at uma.es> wrote:
>>
>>> Thanks Mark for your answer.
>>>
>>> I respond "between lines":
>>>
>>> Mark Wilkinson wrote:
>>>> On Mon, 10 Nov 2008 08:54:32 -0800, Sergio Ramirez Ramirez 
>>>> <srramirez at uma.es> wrote:
>>>>
>>>>
>>>>>  * Timeout: If the time limit is got when trying to connect to the
>>>>>      service.
>>>>>    * Disconnected: The connection with the service couldn't be 
>>>>> established.
>>>>
>>>> What's the difference between these two?
>>>>
>>> Really disconnected is more general than Timeout. It can be used 
>>> when is not possible to connect with the service but not exactly due 
>>> a timeout.
>>>>
>>>>
>>>>> As you can see Eddie, the proposal is compatible with the one you 
>>>>> mention. The only difference is the way to query the status of the 
>>>>> service, that know can  be accessible using the functions of the 
>>>>> MOBY Central..
>>>>
>>>>
>>>> Thanks Sergio!  As usual, the INB has put together a very clear and 
>>>> thought-out proposal, and it is perfectly consistent with what we 
>>>> had in mind!  fantastic!
>>>>
>>>> If I understand correctly, you currently keep this kind of metadata 
>>>> in the INB extended MOBY Central, right?  How tightly integrated is 
>>>> it with MOBY Central?  (i.e. have you changed the MOBY Central API 
>>>> such that you query for this metadata inside of the functions 
>>>> defined by the 'official' API, or is this data accessible via new 
>>>> API functions?)
>>> The idea is to make the information of the test done accessible 
>>> thought the API. For that  we need to include the status and the 
>>> last test done as new information of the registry. Related with the 
>>> functions, the proposal was to extend the functions to recover the 
>>> list of services so they returns also information about the status. 
>>> Also was proposed that those services that don't pass the test 
>>> weren't returned in the list by default so the users always got a 
>>> list of working services
>>>>
>>>> We (my lab) are similarly interested in QoS issues, and where to 
>>>> store this kind of QoSD metadata.  We're looking at repositories 
>>>> such as BioCatalogue as a possibile store, since we don't think 
>>>> that this is the kind of metadata that the registry itself should 
>>>> be in control of (in fact, the registry may not even be capable of 
>>>> knowing!)
>>>> Do you or your colleagues at INB have strong opinions about where 
>>>> to store this metadata, and what the API should be to query for it?
>>> In my personal opinion the information have to be stored in 
>>> accesible way (web-service, xml or similar) so can be handle 
>>> programmability. The real format and protocol used for stored them 
>>> is not the most important thing.
>>>>
>>>> best wishes!
>>>>
>>>> Mark
>>>>
>>>> _______________________________________________
>>>> MOBY-dev mailing list
>>>> MOBY-dev at lists.open-bio.org
>>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>>
>>>
>>>
>>
>>
>>
>> -- 
>> Mark D Wilkinson, PI Bioinformatics
>> Assistant Professor, Medical Genetics
>> The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
>> Providence Heart + Lung Institute
>> University of British Columbia - St. Paul's Hospital
>> Vancouver, BC, Canada
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 


From groscurt at mpiz-koeln.mpg.de  Wed Nov 12 09:36:02 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 12 Nov 2008 15:36:02 +0100
Subject: [MOBY-dev] Compilation error in jmoby
Message-ID: <491AE9D2.9070804@mpiz-koeln.mpg.de>

Hi,

i just made an update of the jmoby and i get several errors concerning

Unhandled exception type NoSuccessException

in
moby/src/main/org/biomoby/client/GraphsServlet
moby/src/main/org/biomoby/service/dashboard/RegistryModel
moby/src/main/org/biomoby/service/generator/DataTypesGenerator
moby/src/main/org/biomoby/service/generator/ServicesGenerator

they apparently all coming from the getDataTypes method in the central 
classes.

Cheers
andreas

-- 
=====================================================
  Dipl. Bioinf. Andreas Groscurth
  Bioinformatics Software Developer
  Plant Computational Biology group
  Max-Planck Institute for plant breeding research
  Carl-von-Linne Weg 10
  50829 Cologne
  Germany
  +49(0) 221 5062449
=====================================================


From martin.senger at gmail.com  Wed Nov 12 09:48:48 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Wed, 12 Nov 2008 20:18:48 +0530
Subject: [MOBY-dev] Compilation error in jmoby
In-Reply-To: <491AE9D2.9070804@mpiz-koeln.mpg.de>
References: <491AE9D2.9070804@mpiz-koeln.mpg.de>
Message-ID: <4d93f07c0811120648x55a2d8bctc206a85ddd5afc3e@mail.gmail.com>

No, I cannot confirm the error. Everything works fine in my local copy. I
have 'cvs update -dP', and make 'ant clean compile' and no error reported,
compilation successful.

Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger

From edward.kawas at gmail.com  Wed Nov 12 09:49:53 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 12 Nov 2008 06:49:53 -0800
Subject: [MOBY-dev] Compilation error in jmoby
In-Reply-To: <491AE9D2.9070804@mpiz-koeln.mpg.de>
References: <491AE9D2.9070804@mpiz-koeln.mpg.de>
Message-ID: <491aed18.1f588c0a.2404.fffffcc7@mx.google.com>

I am not having any problems compiling.

What are the exact errors?

Eddie

-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Andreas Groscurth
Sent: November-12-08 6:36 AM
To: Core developer announcements
Subject: [MOBY-dev] Compilation error in jmoby

Hi,

i just made an update of the jmoby and i get several errors concerning

Unhandled exception type NoSuccessException

in
moby/src/main/org/biomoby/client/GraphsServlet
moby/src/main/org/biomoby/service/dashboard/RegistryModel
moby/src/main/org/biomoby/service/generator/DataTypesGenerator
moby/src/main/org/biomoby/service/generator/ServicesGenerator

they apparently all coming from the getDataTypes method in the central 
classes.

Cheers
andreas

-- 
=====================================================
  Dipl. Bioinf. Andreas Groscurth
  Bioinformatics Software Developer
  Plant Computational Biology group
  Max-Planck Institute for plant breeding research
  Carl-von-Linne Weg 10
  50829 Cologne
  Germany
  +49(0) 221 5062449
=====================================================

_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev


From groscurt at mpiz-koeln.mpg.de  Wed Nov 12 10:01:22 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 12 Nov 2008 16:01:22 +0100
Subject: [MOBY-dev] Compilation error in jmoby
In-Reply-To: <491aed18.1f588c0a.2404.fffffcc7@mx.google.com>
References: <491AE9D2.9070804@mpiz-koeln.mpg.de>
	<491aed18.1f588c0a.2404.fffffcc7@mx.google.com>
Message-ID: <491AEFC2.4060908@mpiz-koeln.mpg.de>

then i guess something with my eclipse is broken.

in the Central class there is the method "getDataTypes"  which throws an 
NoSuccessException.
All methods which call that do not catch that exception or throw them.

Thats at least my local problem here ;-)

Edward Kawas wrote:
> I am not having any problems compiling.
>
> What are the exact errors?
>
> Eddie
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Andreas Groscurth
> Sent: November-12-08 6:36 AM
> To: Core developer announcements
> Subject: [MOBY-dev] Compilation error in jmoby
>
> Hi,
>
> i just made an update of the jmoby and i get several errors concerning
>
> Unhandled exception type NoSuccessException
>
> in
> moby/src/main/org/biomoby/client/GraphsServlet
> moby/src/main/org/biomoby/service/dashboard/RegistryModel
> moby/src/main/org/biomoby/service/generator/DataTypesGenerator
> moby/src/main/org/biomoby/service/generator/ServicesGenerator
>
> they apparently all coming from the getDataTypes method in the central 
> classes.
>
> Cheers
> andreas
>
>   


-- 
=====================================================
  Dipl. Bioinf. Andreas Groscurth
  Bioinformatics Software Developer
  Plant Computational Biology group
  Max-Planck Institute for plant breeding research
  Carl-von-Linne Weg 10
  50829 Cologne
  Germany
  +49(0) 221 5062449
=====================================================


From edward.kawas at gmail.com  Wed Nov 12 15:06:59 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 12 Nov 2008 12:06:59 -0800
Subject: [MOBY-dev] moby unit testing
Message-ID: <491b376a.2a528c0a.1c17.15fa@mx.google.com>

Hi,

Mark and I have made available the new panel for, the webstart version of,
Dashboard (http://moby.ucalgary.ca/dashboard/) that can be used for creating
unit tests for Biomoby services. 

If you are at all interested, please try it out and let the list what you
think about it.

Thanks,

Eddie


From skhadar at gmail.com  Fri Nov 14 08:09:32 2008
From: skhadar at gmail.com (Shameer Khadar)
Date: Fri, 14 Nov 2008 18:39:32 +0530
Subject: [MOBY-dev] Reg. WebServices
Message-ID: <b6ff81950811140509o462e2c41rca21b0dfb9ccf447@mail.gmail.com>

Dear MOBY team,

I recently worked on an integrated bioinformatics  server. Currently the
servers is running on a single node CentOS box. Several users wanted to run
this using multiple queries, hence we would like to extend this as a Web
Service (SOAP/REST based), Can you guys tell me how can I deploy it using
MOBY. Is there any best practices guide to start development of webservices
?

Many thanks in advance,
-- 
K. Shameer
------------------------------------------------------------
Computational Biology Lab, NCBS - TIFR
Molecular Biophysics Unit, IISc

From srramirez at uma.es  Fri Nov 14 09:45:52 2008
From: srramirez at uma.es (Sergio Ramirez Ramirez)
Date: Fri, 14 Nov 2008 15:45:52 +0100
Subject: [MOBY-dev] Reg. WebServices
In-Reply-To: <b6ff81950811140509o462e2c41rca21b0dfb9ccf447@mail.gmail.com>
References: <b6ff81950811140509o462e2c41rca21b0dfb9ccf447@mail.gmail.com>
Message-ID: <491D8F20.4090403@uma.es>

Hello Shameer

In the INB we have some tutorials about how to deploy MOBY Services 
please take a look of this link 
(http://chirimoyo.ac.uma.es/wiki/index.php/MOWServ:Registration) and 
feel free of ask anything you don't understand.

Best Regards, Sergio


Shameer Khadar wrote:
> Dear MOBY team,
>
> I recently worked on an integrated bioinformatics  server. Currently the
> servers is running on a single node CentOS box. Several users wanted to run
> this using multiple queries, hence we would like to extend this as a Web
> Service (SOAP/REST based), Can you guys tell me how can I deploy it using
> MOBY. Is there any best practices guide to start development of webservices
> ?
>
> Many thanks in advance,
>   


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 


From edward.kawas at gmail.com  Fri Nov 14 13:20:16 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Fri, 14 Nov 2008 10:20:16 -0800
Subject: [MOBY-dev] Reg. WebServices
In-Reply-To: <491D8F20.4090403@uma.es>
References: <b6ff81950811140509o462e2c41rca21b0dfb9ccf447@mail.gmail.com>
	<491D8F20.4090403@uma.es>
Message-ID: <491dc16e.16538c0a.2bc7.ffffb587@mx.google.com>

Hi,

If you are interested in how to go about developing services in perl:
http://search.cpan.org/dist/MOSES-MOBY/

Or Java:

http://biomoby.open-bio.org/CVS_CONTENT/moby-live/Java/docs/Moses.html

Hope this helps,

Eddie

-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Sergio Ramirez
Ramirez
Sent: November-14-08 6:46 AM
To: Core developer announcements
Subject: Re: [MOBY-dev] Reg. WebServices

Hello Shameer

In the INB we have some tutorials about how to deploy MOBY Services 
please take a look of this link 
(http://chirimoyo.ac.uma.es/wiki/index.php/MOWServ:Registration) and 
feel free of ask anything you don't understand.

Best Regards, Sergio


Shameer Khadar wrote:
> Dear MOBY team,
>
> I recently worked on an integrated bioinformatics  server. Currently the
> servers is running on a single node CentOS box. Several users wanted to
run
> this using multiple queries, hence we would like to extend this as a Web
> Service (SOAP/REST based), Can you guys tell me how can I deploy it using
> MOBY. Is there any best practices guide to start development of
webservices
> ?
>
> Many thanks in advance,
>   


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 

_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev



From groscurt at mpiz-koeln.mpg.de  Wed Nov 19 10:37:35 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 19 Nov 2008 16:37:35 +0100
Subject: [MOBY-dev] Dashboard as Webstart
Message-ID: <492432BF.8040809@mpiz-koeln.mpg.de>

Hi all,

as you might know I have created a Java Webstart version of the 
Dashboard which is available already on Biomoby.org.

I have now added some ant tasks in the dashboardBuild.xml file to enable 
the creation of the Dashboard Webstart for everyone and checked it into 
the cvs.

You can now create your own Dashboard Webstart by typing 'ant 
dashboard-webstart'.

You will be guided through this creation process and you also can find 
the documentation in docs/DashboardWebstart.html.

For any bugs, errors or comments (I'm not sure if the ant code is way 
too complex) feel free to post here.

Cheers
Andreas

From julien.wollbrett at cirad.fr  Tue Nov 25 11:12:49 2008
From: julien.wollbrett at cirad.fr (Julien Wollbrett)
Date: Tue, 25 Nov 2008 17:12:49 +0100
Subject: [MOBY-dev] integration of datatypes with multiparental relations in
 the datatype ontology
Message-ID: <492C2401.7070003@cirad.fr>

I made a plugin to automatically integrate an OWL ontology into the 
datatype ontology.
It's a plugin for prot?g? in which we need to specify both endpoint and
namespace of the specified central registry.
This plugin run with a "simple" ontology (i.e. ontology with no
multiparental relations).
The problem is that I can't create a datatype with multiparental relation.
A lot of biological ontologies have multiparental relations. How can I
integrate these ontologies into the datatype ontology of a specified
central registry?
Did you planned to integrate multiparental relations in biomoby?
Thank you for your help

Julien Wollbrett

From gordonp at ucalgary.ca  Tue Nov 25 13:59:45 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Tue, 25 Nov 2008 11:59:45 -0700
Subject: [MOBY-dev] integration of datatypes with multiparental
 relations in the datatype ontology
In-Reply-To: <492C2401.7070003@cirad.fr>
References: <492C2401.7070003@cirad.fr>
Message-ID: <492C4B21.9050403@ucalgary.ca>

Hi Julien,

The short answer is No. :-)

Can you present a compelling case for multiple inheritance of a 
datatype?  The typical arguments for objects (which have methods, not 
just data) don't apply...

Julien Wollbrett wrote:
> I made a plugin to automatically integrate an OWL ontology into the 
> datatype ontology.
> It's a plugin for prot?g? in which we need to specify both endpoint and
> namespace of the specified central registry.
> This plugin run with a "simple" ontology (i.e. ontology with no
> multiparental relations).
> The problem is that I can't create a datatype with multiparental 
> relation.
> A lot of biological ontologies have multiparental relations. How can I
> integrate these ontologies into the datatype ontology of a specified
> central registry?
> Did you planned to integrate multiparental relations in biomoby?
> Thank you for your help
>
> Julien Wollbrett
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
>

From markw at illuminae.com  Tue Nov 25 14:20:27 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Tue, 25 Nov 2008 11:20:27 -0800
Subject: [MOBY-dev] [moby] integration of datatypes with multiparental
 relations in the datatype ontology
In-Reply-To: <492C2401.7070003@cirad.fr>
References: <492C2401.7070003@cirad.fr>
Message-ID: <op.uk6v0dhnnbznux@mark.ubic.ubc.ca>

Hi Julien,

I don't see how it would be possible to integrate an "arbitrary" OWL  
ontology into the datatype (you mean Object, right?) ontology in any  
case... Can you please explain in more detail?

At this time, the ISA relationship in the Moby object ontology only allows  
single parenting and there are no plans to change that at the moment.   
This is because the serialization of instances of Moby Objevt nodes is  
tied to the ISA relationship, so it would be a fairly significant change  
in "logic" to allow multiple parenting I suspect...

Mark



On Tue, 25 Nov 2008 08:12:49 -0800, Julien Wollbrett  
<julien.wollbrett at cirad.fr> wrote:

> I made a plugin to automatically integrate an OWL ontology into the  
> datatype ontology.
> It's a plugin for prot?g? in which we need to specify both endpoint and
> namespace of the specified central registry.
> This plugin run with a "simple" ontology (i.e. ontology with no
> multiparental relations).
> The problem is that I can't create a datatype with multiparental  
> relation.
> A lot of biological ontologies have multiparental relations. How can I
> integrate these ontologies into the datatype ontology of a specified
> central registry?
> Did you planned to integrate multiparental relations in biomoby?
> Thank you for your help
>
> Julien Wollbrett
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev



From groscurt at mpiz-koeln.mpg.de  Wed Nov 26 06:19:00 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 26 Nov 2008 12:19:00 +0100
Subject: [MOBY-dev] New Mobyclient
Message-ID: <492D30A4.1030005@mpiz-koeln.mpg.de>

HI all,

I just checked in a new moby client called FeatureClient. For those of 
you who know my Jabba application 
(http://bioinfo.mpiz-koeln.mpg.de/jabba) - basically its the client i 
use there.

Its a usual moby client with some features:

- Calling services with the concrete objects (like AminoAcidSequence) 
instead of the internal Moby API structures. This helps to understand 
the input and output handling better and to work with full data structures.

- Service discovery based on service name or input/output definition 
(like return all services which consume an AGI locus code and return 
PMIDs). This enables to automatically detect group of services which 
fulfill identical or nearly the same tasks.

- Parallelization for multiple service calls. The client enables you to 
call multiple services at the same time.

- Using a timeout if the service does not return in that time. Limit the 
time you want to wait for a service to return by providing a maximal 
time in which the service shall give back a result.

- Calling a service once with possible different inputs. If you want to 
call one (or more) service(s) with several inputs the client does help you.

if you update your moby you will find more information and how to 
build/use it in the html documentation files and in the README file in 
moby-live/Java/src/support/featureClient

For any concerns,comments or bugs please contact me

Cheers
Andreas


From groscurt at mpiz-koeln.mpg.de  Fri Nov 28 07:08:41 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Fri, 28 Nov 2008 13:08:41 +0100
Subject: [MOBY-dev] Secure (Java) Biomoby web services
Message-ID: <492FDF49.6000702@mpiz-koeln.mpg.de>

Hi,

I have committed changes in the jMoby code to allow people to access 
secure BioMoby web services (this update does not inflict the existing 
code). I also set up a tutorial if someone is interested in creating 
and/or calling a BioMoby secure web service 
(http://www.eu-sol.net/science/bioinformatics/tutorials/secure-biomoby-web-services).

Unfortunately the current tutorial / implementation is only valid for 
Java. It would be great if a Perl person could have a look at it and 
contact me so that we can also set up the system for Perl (the system 
follows basically a usal SSL & HTTP authentication principle).

If you find any errors or have any comments, questions please contact me.

Thanks and cheers
Andreas

-- 
/***************************************************
  Dipl. Bioinf. Andreas Groscurth
  Software developer
  Plant Computational Biology group
  Max-Planck Institute for plant breeding research
  Carl-von-Linne Weg 10
  50829 Cologne
  Germany
  +49(0) 221 5062449
***************************************************/


From dmitry.repchevski at bsc.es  Fri Nov 28 17:40:39 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Fri, 28 Nov 2008 14:40:39 -0800
Subject: [MOBY-dev] Secure (Java) Biomoby web services
In-Reply-To: <492FDF49.6000702@mpiz-koeln.mpg.de>
References: <492FDF49.6000702@mpiz-koeln.mpg.de>
Message-ID: <49307367.7000202@bsc.es>

Hello Andreas,

Last INB meeting (in Malaga) we discussed it too.
The good thing would be to use SSL without a certificate, unfortunately, 
I didn't find a way to configure JBoss in a such way... (?)

The idea is that user don't need to have a certificate to connect to a 
server, but at the same time use SSL encryption to protect its data.

Unfortunately, in your examples I didn't find any configuration of SSL 
on the server side...

In JBoss the configuration is done using
$JBOSS_HOME$\server\$PROFILE$\deploy\jboss-web.deployer\server.xml

*********************************************************
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS"

               ciphers="SSL_DH_anon_WITH_RC4_128_MD5
                        TLS_DH_anon_WITH_AES_128_CBC_SHA
                        TLS_DH_anon_WITH_AES_256_CBC_SHA
                        SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
                        SSL_DH_anon_WITH_DES_CBC_SHA
                        TLS_ECDH_anon_WITH_RC4_128_SHA
                        TLS_ECDH_anon_WITH_AES_128_CBC_SHA
                        TLS_ECDH_anon_WITH_AES_256_CBC_SHA
                        TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA
                        SSL_DH_anon_EXPORT_WITH_RC4_40_MD5
                        SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA
                        TLS_ECDH_anon_WITH_NULL_SHA"

               keystoreFile="${jboss.server.home.dir}/conf/jboss.key"
               keystorePass="PASSWORD_HERE"
               />
*********************************************************

Note the list of supported ciphers (the list is for JDK6, JDK5 has a 
smaller list) - they are anonymous.

The problem is that even SSL allows certificate-free communication 
(using any of those ciphers) JBoss (Tomcat) FAILS if I do not specify a 
keystore...
At the same time on a client side, a client receives a certificate from 
JBoss and fails to find it (obviously).

Another comment is that in your web.xml you don't protect the service 
using SSL, to do so you must specify

***************************************************
<user-data-constraint>
     <transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
***************************************************

I would be happy someone would find a way of using SLL WITHOUT a 
certificates, because you can not expect a user of, say, an applet to 
know how to import them using a keytool.

Kind regards,

Dmitry


From groscurt at mpiz-koeln.mpg.de  Fri Nov 28 09:29:11 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Fri, 28 Nov 2008 15:29:11 +0100
Subject: [MOBY-dev] Secure (Java) Biomoby web services
In-Reply-To: <49307367.7000202@bsc.es>
References: <492FDF49.6000702@mpiz-koeln.mpg.de> <49307367.7000202@bsc.es>
Message-ID: <49300037.3040700@mpiz-koeln.mpg.de>

Hi Dimitry,

thanks for your comments. Of course it would be great to have a SSL 
connection without the problem of having and managing the certificates. 
In my tutorial i dont explain the configuration of SSL as this can be 
found on the internet and I dont propose the way of a certificate-less SSL.
But its an interesting idea and I will try to see if I find any solution 
for this and let you know if i find something.... and please keep me 
uptodate if you find something :)


Dmitry Repchevsky wrote:
> Another comment is that in your web.xml you don't protect the service 
> using SSL, to do so you must specify
>
> ***************************************************
> <user-data-constraint>
>     <transport-guarantee>CONFIDENTIAL</transport-guarantee>
> </user-data-constraint>
> *************************************************** 
thanks for pointing that out.

Cheers
Andreas

-- 
/***************************************************
  Dipl. Bioinf. Andreas Groscurth
  Software developer
  Plant Computational Biology group
  Max-Planck Institute for plant breeding research
  Carl-von-Linne Weg 10
  50829 Cologne
  Germany
  +49(0) 221 5062449
***************************************************/


From gordonp at ucalgary.ca  Fri Nov 28 14:38:56 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Fri, 28 Nov 2008 12:38:56 -0700
Subject: [MOBY-dev] jars archive?
Message-ID: <493048D0.8070604@ucalgary.ca>

Hi all,

I think that updates to the jars archive accessible from the Web don't 
work anymore (and perhaps not for some time).  I committed a WAR file 
there last month, and notice it isn't at the Web location for the jars 
archive anymore.  i.e. I was expecting there to be a ...

http://biomoby.org/jmoby-jars/jars-archive/SAWSDLServlet.war

Is the jars-archive update from CVS intentionally defunct?  If so, where 
in CVS should we put all the WARs I create for people to write services?

-Paul

From martin.senger at gmail.com  Sat Nov 29 05:01:50 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Sat, 29 Nov 2008 10:01:50 +0000
Subject: [MOBY-dev] jars archive?
In-Reply-To: <493048D0.8070604@ucalgary.ca>
References: <493048D0.8070604@ucalgary.ca>
Message-ID: <4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>

> I think that updates to the jars archive accessible from the Web don't work
> anymore (and perhaps not for some time).


It was not used since jMoby started to use Maven repository instead.


> Is the jars-archive update from CVS intentionally defunct?


Yes, it is defunct - I have stopped a cronjob that runs the update a while
ago.


>  If so, where in CVS should we put all the WARs I create for people to
> write services?


The question is not much where to put it but how to put it :-) These war
file should end up in the location you mentioned (
http://biomoby.org/jmoby-jars/jars-archive/)<http://biomoby.org/jmoby-jars/jars-archive/SAWSDLServlet.war>-
the location still exists and is accessible from outside.

How do you want to proceed now?

Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger

From gordonp at ucalgary.ca  Sat Nov 29 09:53:09 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Sat, 29 Nov 2008 07:53:09 -0700
Subject: [MOBY-dev] jars archive?
In-Reply-To: <4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>
References: <493048D0.8070604@ucalgary.ca>
	<4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>
Message-ID: <49315755.5050102@ucalgary.ca>

I don't care where I have to put it in CVS, but some directory should be 
copied over to that server regularly.  These WARs are updated not 
infrequently, so a manual copying is not really an option.  Can you 
restore some cron job and let me know where I can put them?

Thanks,

Paul
>>  If so, where in CVS should we put all the WARs I create for people to
>> write services?
>>     
>
>
> The question is not much where to put it but how to put it :-) These war
> file should end up in the location you mentioned (
> http://biomoby.org/jmoby-jars/jars-archive/)<http://biomoby.org/jmoby-jars/jars-archive/SAWSDLServlet.war>-
> the location still exists and is accessible from outside.
>
> How do you want to proceed now?
>
> Martin
>
>   

From martin.senger at gmail.com  Sat Nov 29 12:37:05 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Sat, 29 Nov 2008 17:37:05 +0000
Subject: [MOBY-dev] jars archive?
In-Reply-To: <49315755.5050102@ucalgary.ca>
References: <493048D0.8070604@ucalgary.ca>
	<4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>
	<49315755.5050102@ucalgary.ca>
Message-ID: <4d93f07c0811290937v2a226293w405b37fbffc6e1e3@mail.gmail.com>

> I don't care where I have to put it in CVS


Why should I care? :-)


> Can you restore some cron job and let me know where I can put them?


How often do you want to run the cronjob? What do you want to copy - just
*.war files?

Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger

From gordonp at ucalgary.ca  Sat Nov 29 12:39:59 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Sat, 29 Nov 2008 10:39:59 -0700
Subject: [MOBY-dev] jars archive?
In-Reply-To: <4d93f07c0811290937v2a226293w405b37fbffc6e1e3@mail.gmail.com>
References: <493048D0.8070604@ucalgary.ca>	<4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>	<49315755.5050102@ucalgary.ca>
	<4d93f07c0811290937v2a226293w405b37fbffc6e1e3@mail.gmail.com>
Message-ID: <49317E6F.2060706@ucalgary.ca>

Yeah, just the WAR files, where they are now, once a day would be fine.
>
>
> How often do you want to run the cronjob? What do you want to copy - just
> *.war files?
>
> Martin
>
>   

From martin.senger at gmail.com  Sun Nov 30 05:40:48 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Sun, 30 Nov 2008 10:40:48 +0000
Subject: [MOBY-dev] jars archive?
In-Reply-To: <49317E6F.2060706@ucalgary.ca>
References: <493048D0.8070604@ucalgary.ca>
	<4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>
	<49315755.5050102@ucalgary.ca>
	<4d93f07c0811290937v2a226293w405b37fbffc6e1e3@mail.gmail.com>
	<49317E6F.2060706@ucalgary.ca>
Message-ID: <4d93f07c0811300240m5b4f746bme92feb9da37752e2@mail.gmail.com>

> Yeah, just the WAR files, where they are now, once a day would be fine.


Done. Check here http://biomoby.org/jmoby-jars/jars-archive/ to see what is
available from the jar-archives at the moment.

BTW, for the other developers: If you need some files to be available via
regular HTTP - but you wish to have them also under CVS, you can also use
this 'jar-archive' CVS module. The files put there are passed twice a day to
the URL shown above.

Cheers,
Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger

From kebil at psb.ugent.be  Tue Nov  4 13:02:49 2008
From: kebil at psb.ugent.be (Kenny Billiau)
Date: Tue, 4 Nov 2008 14:02:49 +0100 (CET)
Subject: [MOBY-dev] XML parsing takes forever
In-Reply-To: <490a1b7c.0fba720a.690e.4995@mx.google.com>
References: <Pine.LNX.4.64.0810301431300.27319@midas.psb.ugent.be>
	<4909c0c3.27b38c0a.1e6e.fffff90d@mx.google.com>
	<Pine.LNX.4.64.0810301541530.27319@midas.psb.ugent.be>
	<4909ff61.27f8720a.750d.76c6@mx.google.com>
	<Pine.LNX.4.64.0810301950170.27319@midas.psb.ugent.be>
	<490a03ee.22ba720a.17d1.ffff9006@mx.google.com>
	<490A10F9.5050408@cnio.es>
	<490a1b7c.0fba720a.690e.4995@mx.google.com>
Message-ID: <Pine.LNX.4.64.0811041125310.27319@midas.psb.ugent.be>

Hi,

I've run a little testprogram to see where most of the time is spent. I use a stripped down version of the xml file I've sent you. All but a few ElementContainers and no sequence. Just to keep the processing time low enough to debug.

      use MOSES::MOBY::Parser;

      open(FILE, '<', $ARGV[0]);
      my @lines = <FILE>;
      close FILE;

      my $parser = MOSES::MOBY::Parser->new();
      print $parser->parse( method => 'string', data => join(q{}, @lines) );


[davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
[davebowman at hal2000 ~]$ dprofpp
Exporter::Heavy::heavy_export has 14 unstacked calls in outer
Config::Simple::import_from has 1 unstacked calls in outer
utf8::AUTOLOAD has -1 unstacked calls in outer
Config::Simple::verbose has 1 unstacked calls in outer
Config::Simple::vars has 1 unstacked calls in outer
AutoLoader::AUTOLOAD has -3 unstacked calls in outer
Fcntl::AUTOLOAD has -6 unstacked calls in outer
Fcntl::__ANON__ has 6 unstacked calls in outer
Exporter::export has -14 unstacked calls in outer
utf8::SWASHNEW has 1 unstacked calls in outer
Total Elapsed Time = 8.470220 Seconds
   User+System Time = 7.750220 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  68.1   5.280  5.280  69736   0.0001 0.0001  IO::Scalar::print
  6.94   0.538  0.783  64088   0.0000 0.0000  Dumpvalue::stringify
  6.04   0.468  7.173  33755   0.0000 0.0002  Dumpvalue::unwrap
  4.30   0.333  0.333  47091   0.0000 0.0000  File::Spec::Unix::canonpath
  2.43   0.188  7.157  33006   0.0000 0.0002  Dumpvalue::DumpElem
  2.13   0.165  0.165  59776   0.0000 0.0000  overload::mycan
  1.78   0.138  0.224  15990   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
  1.42   0.110  0.708    539   0.0002 0.0013  MOSES::MOBY::Cache::Central::_createDataTypeFromXML
  1.39   0.108  0.108 142023   0.0000 0.0000  UNIVERSAL::isa
  1.29   0.100  0.100    208   0.0005 0.0005  MOSES::MOBY::Cache::Registries::init
  1.20   0.093  0.357  17359   0.0000 0.0000  overload::OverloadedStringify
  1.16   0.090  0.090  30407   0.0000 0.0000  XML::LibXML::Node::nodeName
  1.02   0.079  0.079    539   0.0001 0.0001  MOSES::MOBY::Cache::Central::_clean
  0.99   0.077  0.418    207   0.0004 0.0020  MOSES::MOBY::Generators::Utils::find_file
  0.94   0.073  0.203   3147   0.0000 0.0001  XML::LibXML::Element::getChildrenByTagName

It's fairly obivous now ;)

If I remove the overload operator in MOSES::MOBY::Base, time is reduced 
15fold! But for the 250KB file I've sent, it still takes us a minute to 
parse.

If I simply parse the file with XML::LibXML and no callbacks or custom 
handlers, it's parsed instantly:

[davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
[kebil at psbdev01 ~]$ dprofpp
Total Elapsed Time = 0.079034 Seconds
   User+System Time = 0.079034 Seconds


-Kenny, slowly getting to the bottom of this ;)


On Thu, 30 Oct 2008, Edward Kawas wrote:

> Now I can confirm that it is slow parsing and not an infinite loop! The XML
> file took over an hour on my virtual machine to parse. I will see what I can
> do with other SAX parsers for Perl.
>
> Thanks,
>
> Eddie
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Jos? Mar?a
> Fern?ndez Gonz?lez
> Sent: October-30-08 12:55 PM
> To: Core developer announcements
> Subject: Re: [MOBY-dev] XML parsing takes forever
>
> Hi everybody,
> 	I'm using the XML::LibXML SAX parser for other projects and tasks,
> and it can
> be really, really slow. You don't realize that until you are parsing a large
> file from a pipe, and you see that most of the time the program which is
> parsing is CPU (and not I/O) bounded.
>
> 	Best Regards,
> 		Jos? Mar?a
>
> Edward Kawas wrote:
>> I think that you are right about the parser not going into an infinite
> loop.
>> For some reason, the SAX parser seems to be really slow (although, I don't
>> want to say that until I know for sure).
>>
>> Eddie
>>
>> -----Original Message-----
>> From: moby-dev-bounces at lists.open-bio.org
>> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
>> Sent: October-30-08 11:56 AM
>> To: Core developer announcements
>> Subject: Re: [MOBY-dev] XML parsing takes forever
>>
>> Hi,
>>
>> thx for helping to investigate the problem!
>>
>> I'm pretty sure the parser gets out of it's mess. When you truncate the
>> XML file by removing all but a few ElementContainers, it takes awhile, but
>
>> it actually returns without error.
>>
>> And memory consuption doesn't seem to be a problem either, as mostly 'top'
>
>> reveals that the invoked webservice script uses a steady amount of memory.
>>
>> thx,
>> Kenny
>>
>> On Thu, 30 Oct 2008, Edward Kawas wrote:
>>
>>> So this one might take some time... from what I can see, our parser goes
>>> into an infinite loop (although, I am not certain about that ...).
>>>
>>> I will play around with this XML file and get back to you.
>>>
>>> Thanks,
>>>
>>> Eddie
>>>
>>> -----Original Message-----
>>> From: moby-dev-bounces at lists.open-bio.org
>>> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
>>> Sent: October-30-08 8:03 AM
>>> To: Core developer announcements
>>> Subject: Re: [MOBY-dev] XML parsing takes forever
>>>
>>> Hi,
>>>
>>> here you go,
>>>
>>> Kenny
>>>
>>> On Thu, 30 Oct 2008, Edward Kawas wrote:
>>>
>>>> Hi,
>>>>
>>>> This is the right list! Could you send me the XML for your second
>> service?
>>>> Thanks,
>>>>
>>>> Eddie
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>
>>
>
>

-- 

From kenny.billiau at psb.ugent.be  Tue Nov  4 13:03:47 2008
From: kenny.billiau at psb.ugent.be (Kenny Billiau)
Date: Tue, 4 Nov 2008 14:03:47 +0100 (CET)
Subject: [MOBY-dev] XML parsing takes forever
Message-ID: <Pine.LNX.4.64.0811041403160.27319@midas.psb.ugent.be>

Hi,

I've run a little testprogram to see where most of the time is spent. I use a 
stripped down version of the xml file I've sent you. All but a few 
ElementContainers and no sequence. Just to keep the processing time low enough 
to debug.

      use MOSES::MOBY::Parser;

       open(FILE, '<', $ARGV[0]);
       my @lines = <FILE>;
       close FILE;

       my $parser = MOSES::MOBY::Parser->new();
       print $parser->parse( method => 'string', data => join(q{}, @lines) );


[davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
[davebowman at hal2000 ~]$ dprofpp
Exporter::Heavy::heavy_export has 14 unstacked calls in outer
Config::Simple::import_from has 1 unstacked calls in outer
utf8::AUTOLOAD has -1 unstacked calls in outer
Config::Simple:: verbose has 1 unstacked calls in outer
Config::Simple:: vars has 1 unstacked calls in outer
AutoLoader::AUTOLOAD has -3 unstacked calls in outer
Fcntl:: AUTOLOAD has -6 unstacked calls in outer
Fcntl: :__ANON__ has 6 unstacked calls in outer
Exporter::export has -14 unstacked calls in outer
utf8::SWASHNEW has 1 unstacked calls in outer
Total Elapsed Time = 8.470220 Seconds
   User+System Time = 7.750220 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
   68.1   5.280  5.280  69736   0.0001 0.0001  IO::Scalar::print
   6.94   0.538  0.783  64088   0.0000 0.0000  Dumpvalue::stringify
   6.04   0.468  7.173  33755   0.0000 0.0002  Dumpvalue::unwrap
   4.30   0.333  0.333  47091   0.0000 0.0000  File::Spec::Unix::canonpath
   2.43   0.188  7.157  33006   0.0000 0.0002  Dumpvalue::DumpElem
   2.13   0.165  0.165  59776   0.0000 0.0000  overload::mycan
   1.78   0.138  0.224  15990   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
   1.42   0.110  0.708    539   0.0002 0.0013
   MOSES::MOBY::Cache::Central::_createDataTypeFromXML
   1.39   0.108  0.108 142023   0.0000 0.0000  UNIVERSAL::isa
   1.29   0.100  0.100    208   0.0005 0.0005
   MOSES::MOBY::Cache::Registries::init
   1.20   0.093  0.357  17359   0.0000 0.0000  overload::OverloadedStringify
   1.16   0.090  0.090  30407   0.0000 0.0000  XML::LibXML::Node::nodeName
   1.02   0.079  0.079    539   0.0001 0.0001
   MOSES::MOBY::Cache::Central::_clean
   0.99   0.077  0.418    207   0.0004 0.0020
   MOSES::MOBY::Generators::Utils::find_file
   0.94   0.073  0.203   3147   0.0000 0.0001
   XML::LibXML::Element::getChildrenByTagName

It's fairly obivous now ;)

If I remove the overload operator in MOSES::MOBY::Base, time is reduced 15fold! 
But for the 250KB file I've sent, it still takes us a minute to parse.

If I simply parse the file with XML::LibXML and no callbacks or custom 
handlers, it's parsed instantly:

[davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
[kebil at psbdev01 ~]$ dprofpp
Total Elapsed Time = 0.079034 Seconds
    User+System Time = 0.079034 Seconds


-Kenny, slowly getting to the bottom of this ;)


On Thu, 30 Oct 2008, Edward Kawas wrote:

>  Now I can confirm that it is slow parsing and not an infinite loop! The XML
>  file took over an hour on my virtual machine to parse. I will see what I can
>  do with other SAX parsers for Perl.
>
>  Thanks,
>
>  Eddie
>
>  -----Original Message-----
>  From: moby-dev-bounces at lists.open-bio.org
>  [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Jos? Mar?a
>  Fern?ndez Gonz?lez
>  Sent: October-30-08 12:55 PM
>  To: Core developer announcements
>  Subject: Re: [MOBY-dev] XML parsing takes forever
>
>  Hi everybody,
>  	I'm using the XML::LibXML SAX parser for other projects and tasks,
>  and it can
>  be really, really slow. You don't realize that until you are parsing a large
>  file from a pipe, and you see that most of the time the program which is
>  parsing is CPU (and not I/O) bounded.
>
>   Best Regards,
>    Jos? Mar?a
>
>  Edward Kawas wrote:
> >  I think that you are right about the parser not going into an infinite
>  loop.
> >  For some reason, the SAX parser seems to be really slow (although, I don't
> >  want to say that until I know for sure).
> > 
> >  Eddie
> > 
> >  -----Original Message-----
> >  From: moby-dev-bounces at lists.open-bio.org
> >  [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
> >  Sent: October-30-08 11:56 AM
> >  To: Core developer announcements
> >  Subject: Re: [MOBY-dev] XML parsing takes forever
> > 
> >  Hi,
> > 
> >  thx for helping to investigate the problem!
> > 
> >  I'm pretty sure the parser gets out of it's mess. When you truncate the
> >  XML file by removing all but a few ElementContainers, it takes awhile, but
> 
> >  it actually returns without error.
> > 
> >  And memory consuption doesn't seem to be a problem either, as mostly 'top'
> 
> >  reveals that the invoked webservice script uses a steady amount of memory.
> > 
> >  thx,
> >  Kenny
> > 
> >  On Thu, 30 Oct 2008, Edward Kawas wrote:
> > 
> > >  So this one might take some time... from what I can see, our parser goes
> > >  into an infinite loop (although, I am not certain about that ...).
> > > 
> > >  I will play around with this XML file and get back to you.
> > > 
> > >  Thanks,
> > > 
> > >  Eddie
> > > 
> > >  -----Original Message-----
> > >  From: moby-dev-bounces at lists.open-bio.org
> > >  [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
> > >  Sent: October-30-08 8:03 AM
> > >  To: Core developer announcements
> > >  Subject: Re: [MOBY-dev] XML parsing takes forever
> > > 
> > >  Hi,
> > > 
> > >  here you go,
> > > 
> > >  Kenny
> > > 
> > >  On Thu, 30 Oct 2008, Edward Kawas wrote:
> > > 
> > > >  Hi,
> > > > 
> > > >  This is the right list! Could you send me the XML for your second
> >  service?
> > > >  Thanks,
> > > > 
> > > >  Eddie
>>> _______________________________________________
> > >  MOBY-dev mailing list
> > >  MOBY-dev at lists.open-bio.org
> > >  http://lists.open-bio.org/mailman/listinfo/moby-dev
> > > 
> > 
> 
>

-- 

From kebil at psb.ugent.be  Wed Nov  5 15:57:00 2008
From: kebil at psb.ugent.be (Kenny Billiau)
Date: Wed, 5 Nov 2008 16:57:00 +0100 (CET)
Subject: [MOBY-dev] XML parsing takes forever
In-Reply-To: <Pine.LNX.4.64.0811041403160.27319@midas.psb.ugent.be>
References: <Pine.LNX.4.64.0811041403160.27319@midas.psb.ugent.be>
Message-ID: <Pine.LNX.4.64.0811051418290.27319@midas.psb.ugent.be>

Hi,

after some more tests, I've succesfully reduced the parsing time from over 
1 hour 15 minutes to 17 seconds. Which is not ideal yet, but at least workable :)

This is the profiling result for the original environment. The test xml file is the original file I've sent here without the sequence.

[davebowman at hal2000 ]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_no_dna.xml
Trying to parse.. Done.
[davebowman at hal2000 ]$ dprofpp
Total Elapsed Time = 4348.764 Seconds
   User+System Time = 4344.374 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  97.3   4227. 4227.2 399336   0.0011 0.0011  IO::Scalar::print
  0.88   38.06 4357.4 189130   0.0000 0.0023  Dumpvalue::unwrap
  0.80   34.54 51.532 369048   0.0000 0.0000  Dumpvalue::stringify
  0.54   23.63 23.634 463568   0.0000 0.0000  File::Spec::Unix::canonpath
  0.33   14.23 4356.7 188364   0.0000 0.0023  Dumpvalue::DumpElem


First I simply removed the overload operator "" (as_string). I still get the same results and processing time is reduced drastically.
I don't really know why this way of stringifying (complex) perl types is used here. 
Is there a special reason for it? Can't we simply use Data::Dumper (which prints to a $var by default)?

[davebowman at hal2000 ]$ dprofpp
Total Elapsed Time = 35.85114 Seconds
   User+System Time = 33.54114 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  64.5   21.64 21.646 463568   0.0000 0.0000  File::Spec::Unix::canonpath
  22.9   7.698 31.602 231569   0.0000 0.0000  File::Spec::Unix::catfile
  19.7   6.607  6.607   2144   0.0031 0.0031  MOSES::MOBY::Cache::Registries::init
  18.6   6.251 37.772   2143   0.0029 0.0176  MOSES::MOBY::Generators::Utils::find_file
  6.80   2.282 13.448 231569   0.0000 0.0000  File::Spec::Unix::catdir

A lot of calls are made to MOSES::MOBY::Generators::Utils::find_file, 
which in it's turn does a lot of calls with File::Spec. So I simply cache the results.
Time is reduced to 17 seconds:

Total Elapsed Time = 17.34032 Seconds
   User+System Time = 13.20032 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  51.5   6.807  6.807   2144   0.0032 0.0032  MOSES::MOBY::Cache::Registries::init
  9.01   1.190  2.005 162858   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
  4.65   0.614  0.614 310751   0.0000 0.0000  XML::LibXML::Node::nodeName
  4.64   0.612  1.655  32171   0.0000 0.0001  XML::LibXML::Element::getChildrenByTagName
  4.48   0.592  0.592   5515   0.0001 0.0001  MOSES::MOBY::Cache::Central::_clean

More then half of the time still goes to building up the MOBY environment for each processed element in the XML file. Isn't this a bit overkill? ;)
Is there a way to make a MOSES::MOBY factory that can handle the building up of the moby environment just once in the same (mod_perl) call? Or is there a reason this has been avoided?

I've included the patches for the result caching in 
MOSES::MOBY::Generators::Utils::find_file . there's not much to it. My 
version of MOSES::MOBY still is the one uploaded to cpan: Module MOSES::MOBY (E/EK/EKAWAS/MOSES-MOBY-0.86.tar.gz)

happy hacking,
Kenny


On Tue, 4 Nov 2008, Kenny Billiau wrote:

> Hi,
>
> I've run a little testprogram to see where most of the time is spent. I use a 
> stripped down version of the xml file I've sent you. All but a few 
> ElementContainers and no sequence. Just to keep the processing time low 
> enough to debug.
>
>     use MOSES::MOBY::Parser;
>
>       open(FILE, '<', $ARGV[0]);
>       my @lines = <FILE>;
>       close FILE;
>
>       my $parser = MOSES::MOBY::Parser->new();
>       print $parser->parse( method => 'string', data => join(q{}, @lines) );
>
>
> [davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
> [davebowman at hal2000 ~]$ dprofpp
> Exporter::Heavy::heavy_export has 14 unstacked calls in outer
> Config::Simple::import_from has 1 unstacked calls in outer
> utf8::AUTOLOAD has -1 unstacked calls in outer
> Config::Simple::  verbose has 1 unstacked calls in outer
> Config::Simple::  vars has 1 unstacked calls in outer
> AutoLoader::AUTOLOAD has -3 unstacked calls in outer
> Fcntl::  AUTOLOAD has -6 unstacked calls in outer
> Fcntl: :__ANON__ has 6 unstacked calls in outer
> Exporter::export has -14 unstacked calls in outer
> utf8::SWASHNEW has 1 unstacked calls in outer
> Total Elapsed Time = 8.470220 Seconds
>  User+System Time = 7.750220 Seconds
> Exclusive Times
> %Time ExclSec CumulS #Calls sec/call Csec/c  Name
>   68.1   5.280  5.280  69736   0.0001 0.0001  IO::Scalar::print
>   6.94   0.538  0.783  64088   0.0000 0.0000  Dumpvalue::stringify
>   6.04   0.468  7.173  33755   0.0000 0.0002  Dumpvalue::unwrap
>   4.30   0.333  0.333  47091   0.0000 0.0000  File::Spec::Unix::canonpath
>   2.43   0.188  7.157  33006   0.0000 0.0002  Dumpvalue::DumpElem
>   2.13   0.165  0.165  59776   0.0000 0.0000  overload::mycan
>   1.78   0.138  0.224  15990   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
>   1.42   0.110  0.708    539   0.0002 0.0013
>   MOSES::MOBY::Cache::Central::_createDataTypeFromXML
>   1.39   0.108  0.108 142023   0.0000 0.0000  UNIVERSAL::isa
>   1.29   0.100  0.100    208   0.0005 0.0005
>   MOSES::MOBY::Cache::Registries::init
>   1.20   0.093  0.357  17359   0.0000 0.0000  overload::OverloadedStringify
>   1.16   0.090  0.090  30407   0.0000 0.0000  XML::LibXML::Node::nodeName
>   1.02   0.079  0.079    539   0.0001 0.0001
>   MOSES::MOBY::Cache::Central::_clean
>   0.99   0.077  0.418    207   0.0004 0.0020
>   MOSES::MOBY::Generators::Utils::find_file
>   0.94   0.073  0.203   3147   0.0000 0.0001
>   XML::LibXML::Element::getChildrenByTagName
>
> It's fairly obivous now ;)
>
> If I remove the overload operator in MOSES::MOBY::Base, time is reduced 
> 15fold! But for the 250KB file I've sent, it still takes us a minute to 
> parse.
>
> If I simply parse the file with XML::LibXML and no callbacks or custom 
> handlers, it's parsed instantly:
>
> [davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_small.xml
> [kebil at psbdev01 ~]$ dprofpp
> Total Elapsed Time = 0.079034 Seconds
>    User+System Time = 0.079034 Seconds
>
>
> -Kenny, slowly getting to the bottom of this ;)
>
>
> On Thu, 30 Oct 2008, Edward Kawas wrote:
>
>>   Now I can confirm that it is slow parsing and not an infinite loop! The
>>   XML
>>   file took over an hour on my virtual machine to parse. I will see what I
>>   can
>>   do with other SAX parsers for Perl.
>>
>>   Thanks,
>>
>>   Eddie
>>
>>   -----Original Message-----
>>   From: moby-dev-bounces at lists.open-bio.org
>>   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Jos? Mar?a
>>   Fern?ndez Gonz?lez
>>   Sent: October-30-08 12:55 PM
>>   To: Core developer announcements
>>   Subject: Re: [MOBY-dev] XML parsing takes forever
>>
>>   Hi everybody,
>>   	I'm using the XML::LibXML SAX parser for other projects and tasks,
>>   and it can
>>   be really, really slow. You don't realize that until you are parsing a
>>   large
>>   file from a pipe, and you see that most of the time the program which is
>>   parsing is CPU (and not I/O) bounded.
>>
>>    Best Regards,
>>     Jos? Mar?a
>>
>>   Edward Kawas wrote:
>> >   I think that you are right about the parser not going into an infinite
>>   loop.
>> >   For some reason, the SAX parser seems to be really slow (although, I 
>> >   don't
>> >   want to say that until I know for sure).
>> > 
>> >   Eddie
>> > 
>> >   -----Original Message-----
>> >   From: moby-dev-bounces at lists.open-bio.org
>> >   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
>> >   Sent: October-30-08 11:56 AM
>> >   To: Core developer announcements
>> >   Subject: Re: [MOBY-dev] XML parsing takes forever
>> > 
>> >   Hi,
>> > 
>> >   thx for helping to investigate the problem!
>> > 
>> >   I'm pretty sure the parser gets out of it's mess. When you truncate the
>> >   XML file by removing all but a few ElementContainers, it takes awhile, 
>> >   but
>> 
>> >   it actually returns without error.
>> > 
>> >   And memory consuption doesn't seem to be a problem either, as mostly 
>> >   'top'
>> 
>> >   reveals that the invoked webservice script uses a steady amount of 
>> >   memory.
>> > 
>> >   thx,
>> >   Kenny
>> > 
>> >   On Thu, 30 Oct 2008, Edward Kawas wrote:
>> > 
>> > >   So this one might take some time... from what I can see, our parser 
>> > >   goes
>> > >   into an infinite loop (although, I am not certain about that ...).
>> > > 
>> > >   I will play around with this XML file and get back to you.
>> > > 
>> > >   Thanks,
>> > > 
>> > >   Eddie
>> > > 
>> > >   -----Original Message-----
>> > >   From: moby-dev-bounces at lists.open-bio.org
>> > >   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny 
>> > >   Billiau
>> > >   Sent: October-30-08 8:03 AM
>> > >   To: Core developer announcements
>> > >   Subject: Re: [MOBY-dev] XML parsing takes forever
>> > > 
>> > >   Hi,
>> > > 
>> > >   here you go,
>> > > 
>> > >   Kenny
>> > > 
>> > >   On Thu, 30 Oct 2008, Edward Kawas wrote:
>> > > 
>> > > >   Hi,
>> > > > 
>> > > >   This is the right list! Could you send me the XML for your second
>> >   service?
>> > > >   Thanks,
>> > > > 
>> > > >   Eddie
>> > >  _______________________________________________
>> > >   MOBY-dev mailing list
>> > >   MOBY-dev at lists.open-bio.org
>> > >   http://lists.open-bio.org/mailman/listinfo/moby-dev
>> > > 
>> >
>> 
>> 
>
>

-- 
-------------- next part --------------
--- /usr/lib/perl5/site_perl/5.8.5/MOSES/MOBY/Generators/Utils.pm	2008-04-29 21:45:41.000000000 +0200
+++ ../modules/MOSES/MOBY/Generators/Utils.pm	2008-11-05 16:47:23.000000000 +0100
@@ -26,7 +26,7 @@
  # find a file located somewhere in @INC
  use MOSES::MOBY::Generators::Utils;
  my $file = MOSES::MOBY::Generators::Utils->find_file ('resource.file');
- 
+
 =cut
 
 =head1 DESCRIPTION
@@ -61,17 +61,29 @@
 
 =cut
 
+my %full_path_of = ();
+
 sub find_file {
     my ($self, $default_start, @names) = @_;
     my $fixed_part = File::Spec->catfile (@names);
+    return $full_path_of{ $fixed_part } if exists $full_path_of{ $fixed_part };
+
     my $result = File::Spec->catfile ($default_start, $fixed_part);
-    return $result if -e $result;
+    if (-e $result) {
+        $full_path_of{ $fixed_part } = $result;
+        return $result;
+    }
 
     foreach my $idx (0 .. $#INC) {
-	$result = File::Spec->catfile ($INC[$idx], $fixed_part);
-	return $result if -e $result;
+        $result = File::Spec->catfile ($INC[$idx], $fixed_part);
+        if (-e $result) {
+            $full_path_of{ $fixed_part } = $result;
+            return $result;
+        }
     }
-    return File::Spec->catfile ($default_start, $fixed_part);
+    $result = File::Spec->catfile ($default_start, $fixed_part);
+    $full_path_of{ $fixed_part } = $result;
+    return $result;
 }
 
 1;

From dmitry.repchevski at bsc.es  Thu Nov  6 20:58:07 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 06 Nov 2008 12:58:07 -0800
Subject: [MOBY-dev] INB Base64 (binary) type proposal
Message-ID: <49135A5F.20703@bsc.es>

Hello everybody,

These days there were a technical INB meeting in Malaga (Spain) and I 
like to comment some decisions taken there.

Most interesting to me is the introduction of new Moby primitive type 
"Binary" that will correspond to the xml schema:

********************************************************
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:tns="http://www.biomoby.org/moby" 
xmlns="http://www.biomoby.org/moby" 
xmlns:moby="http://www.biomoby.org/moby" 
xmlns:xs="http://www.w3.org/2001/XMLSchema" 
attributeFormDefault="qualified" elementFormDefault="qualified" 
version="1.0" targetNamespace="http://www.biomoby.org/moby">

<xs:element name="Binary" type="MobyBinary"></xs:element>

<xs:complexType name="MobyBinary">
<xs:simpleContent>
 <xs:extension
   base="xs:base64Binary"
   ns1:expectedContentTypes="application/octet-stream"
   xmlns:ns1="http://www.w3.org/2005/05/xmlmime">
 </xs:extension>
</xs:simpleContent>
</xs:complexType>

</xs:schema>
********************************************************

The encoding inside a Moby Message will be a standard base64 coded chunk 
of data:
********************************************************
<moby:Binary id="" namespace="">dGVzdA==</moby:Binary>
********************************************************

The need of new type is enough obvious: we need to pass binary data. 
Today, instead of using a primitive type many service developers mess a 
datatype tree with their own solutions (text-base64, b64_encoded_ppm, 
Zip_encoded, Image_PNG, Image_Encoded, glycoPNG ...)
Even we do not expect that service providers will change their services 
to utilize this new type, it would be very useful for many new types.

At the moment the encoding itself is a responsibility of a developer and 
such a type will free them from the need to encode/decode process.
The implementation is pretty easy and as soon as our key Perl developer 
is back from vacation we expect him to introduce it into the perl moby 
code.

So far this is the official INB proposal and we would be happy to here 
comments from other develops.
Any comments or suggestions are welcome.

Warm regards,

Dmitry

P.S. http://inb.bsc.es/documents/base64bit.pdf



From groscurt at mpiz-koeln.mpg.de  Thu Nov  6 12:45:54 2008
From: groscurt at mpiz-koeln.mpg.de (groscurt)
Date: Thu, 6 Nov 2008 13:45:54 +0100
Subject: [MOBY-dev] INB Base64 (binary) type proposal
In-Reply-To: <49135A5F.20703@bsc.es>
Message-ID: <0bf412a6235d581f18ff007cd76e92e6@mpizmail.mpiz-koeln.mpg.de>

> 
> P.S. http://inb.bsc.es/documents/base64bit.pdf

Object not found for me...

cheers
andreas





From dmitry.repchevski at bsc.es  Thu Nov  6 22:12:01 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Thu, 06 Nov 2008 14:12:01 -0800
Subject: [MOBY-dev] INB Base64 (binary) type proposal
In-Reply-To: <0bf412a6235d581f18ff007cd76e92e6@mpizmail.mpiz-koeln.mpg.de>
References: <0bf412a6235d581f18ff007cd76e92e6@mpizmail.mpiz-koeln.mpg.de>
Message-ID: <49136BB1.90604@bsc.es>

Opps I'm sorry...

http://inb.bsc.es/documents/Base64bit.pdf

uppercase...




From dcicinsain at mmb.pcb.ub.es  Thu Nov  6 13:26:23 2008
From: dcicinsain at mmb.pcb.ub.es (Damjan Cicin-Sain)
Date: Thu, 06 Nov 2008 14:26:23 +0100
Subject: [MOBY-dev] INB Base64 (binary) type proposal
In-Reply-To: <0bf412a6235d581f18ff007cd76e92e6@mpizmail.mpiz-koeln.mpg.de>
References: <0bf412a6235d581f18ff007cd76e92e6@mpizmail.mpiz-koeln.mpg.de>
Message-ID: <1225977983.6568.2.camel@damjan-laptop>

http://inb.bsc.es/documents/Base64bit.pdf

On Thu, 2008-11-06 at 13:45 +0100, groscurt wrote:
> > 
> > P.S. http://inb.bsc.es/documents/base64bit.pdf
> 
> Object not found for me...
> 
> cheers
> andreas
> 
> 
> 
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev



From py at pingoured.fr  Thu Nov  6 16:28:34 2008
From: py at pingoured.fr (Pierre-Yves Chibon)
Date: Thu, 06 Nov 2008 17:28:34 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <48528447.7040807@toulouse.inra.fr>
References: <4850BC9D.8090800@mpiz-koeln.mpg.de>	<485127d8.25bb720a.622e.0d84@mx.google.com>	<48512862.7030104@mpiz-koeln.mpg.de>	<4851431d.1420720a.52bf.243d@mx.google.com>	<48527CD8.9060904@mpiz-koeln.mpg.de>
	<48528447.7040807@toulouse.inra.fr>
Message-ID: <49131B32.2010605@pingoured.fr>

Sebastien Carrere wrote:
> Here is my list of Dead services (Ihave got also list for the 3 past 
> days for testRegistry and Inab registry) :
> http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral


Just by curiosity :
  - do you have a list of working services ?
  - do you have available your results from the testing repository ? (by 
the way, why are you testing it ?)

Thanks in advance,

Regards,

Pierre


From Sebastien.Carrere at toulouse.inra.fr  Thu Nov  6 17:16:47 2008
From: Sebastien.Carrere at toulouse.inra.fr (Sebastien Carrere)
Date: Thu, 06 Nov 2008 18:16:47 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49131B32.2010605@pingoured.fr>
References: <4850BC9D.8090800@mpiz-koeln.mpg.de>	<485127d8.25bb720a.622e.0d84@mx.google.com>	<48512862.7030104@mpiz-koeln.mpg.de>	<4851431d.1420720a.52bf.243d@mx.google.com>	<48527CD8.9060904@mpiz-koeln.mpg.de>	<48528447.7040807@toulouse.inra.fr>
	<49131B32.2010605@pingoured.fr>
Message-ID: <4913267F.8030307@toulouse.inra.fr>

Bonjour
>
> Just by curiosity :
>  - do you have a list of working services ?
Yes, I have:
For produciton registry:
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/list.mobycentral
For testing registry:
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/list.opencentral
For Inab registry:
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/list.Inab
>  - do you have available your results from the testing repository ? 
> (by the way, why are you testing it ?)
Here is the black list for testing registry:
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.opencentral

I test all these registries to filter really available services in 
Remora (you can choose the registry you want among these 3 - I can add 
others if needed).
By default, production registry is used.

Sebastien



-------------- next part --------------
A non-text attachment was scrubbed...
Name: Sebastien_Carrere.vcf
Type: text/x-vcard
Size: 387 bytes
Desc: not available
URL: <http://lists.open-bio.org/pipermail/moby-dev/attachments/20081106/fac42ea4/attachment-0002.vcf>

From serr at ac.uma.es  Thu Nov  6 17:14:00 2008
From: serr at ac.uma.es (Sergio Ramirez Ramirez)
Date: Thu, 06 Nov 2008 18:14:00 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49131B32.2010605@pingoured.fr>
References: <4850BC9D.8090800@mpiz-koeln.mpg.de>	<485127d8.25bb720a.622e.0d84@mx.google.com>	<48512862.7030104@mpiz-koeln.mpg.de>	<4851431d.1420720a.52bf.243d@mx.google.com>	<48527CD8.9060904@mpiz-koeln.mpg.de>	<48528447.7040807@toulouse.inra.fr>
	<49131B32.2010605@pingoured.fr>
Message-ID: <491325D8.5050402@ac.uma.es>

Hello Pierre.

Just in advance. As Dimitry has comment we had a meeting of the INB this 
week, and one of the proposals was about the quality of services.
The solution proposed is to extend the information of the services with 
information about the status of the service (Ok, no tested, no 
responding, etc)

The services without "enough" quality will be hidden from the general 
list giving the user always services that work properly. The hidden one 
will still available if the user ask for them so they can be 
deregistered or update their information for make them back to work.

I'm preparing a more detailed document that I will sent soon.
Best regards
Sergio.

Pierre-Yves Chibon wrote:
> Sebastien Carrere wrote:
>> Here is my list of Dead services (Ihave got also list for the 3 past 
>> days for testRegistry and Inab registry) :
>> http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral 
>>
>
>
> Just by curiosity :
>  - do you have a list of working services ?
>  - do you have available your results from the testing repository ? 
> (by the way, why are you testing it ?)
>
> Thanks in advance,
>
> Regards,
>
> Pierre
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 



From edward.kawas at gmail.com  Thu Nov  6 18:37:55 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Thu, 6 Nov 2008 10:37:55 -0800
Subject: [MOBY-dev] XML parsing takes forever
In-Reply-To: <Pine.LNX.4.64.0811051418290.27319@midas.psb.ugent.be>
References: <Pine.LNX.4.64.0811041403160.27319@midas.psb.ugent.be>
	<Pine.LNX.4.64.0811051418290.27319@midas.psb.ugent.be>
Message-ID: <4913398f.02578c0a.5c4f.ffff8bc1@mx.google.com>

Hi Kevin,

I applied your patch to Utils.pm. I also modified
MOSES::MOBY::Cache::Registries so that a file read isn?t done every time
that module is instantiated. Finally, I moved IO::Scalar to IO::String.

Now, instead of 1 hour+ to parse the XML file, we have got it down to just
over a minute. I am hesitant to remove the as_string overload in Base.pm. We
use it for displaying our datatypes nicely (better than if Data::Dumper was
used). If we can find a way to produce the same information as as_string and
be more efficient, I would swap that in right away.

The changes are in the CVS @ moby-live/Perl/MOSES-MOBY .

Please keep hacking away! I really appreciate all of your help so far!

Thanks.

Eddie



-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny Billiau
Sent: November-05-08 7:57 AM
To: Core developer announcements
Subject: Re: [MOBY-dev] XML parsing takes forever

Hi,

after some more tests, I've succesfully reduced the parsing time from over
1 hour 15 minutes to 17 seconds. Which is not ideal yet, but at least
workable :)

This is the profiling result for the original environment. The test xml file
is the original file I've sent here without the sequence.

[davebowman at hal2000 ]$ /usr/bin/perl -d:DProf mobytest.pl mobysav_no_dna.xml
Trying to parse.. Done.
[davebowman at hal2000 ]$ dprofpp
Total Elapsed Time = 4348.764 Seconds
   User+System Time = 4344.374 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  97.3   4227. 4227.2 399336   0.0011 0.0011  IO::Scalar::print
  0.88   38.06 4357.4 189130   0.0000 0.0023  Dumpvalue::unwrap
  0.80   34.54 51.532 369048   0.0000 0.0000  Dumpvalue::stringify
  0.54   23.63 23.634 463568   0.0000 0.0000  File::Spec::Unix::canonpath
  0.33   14.23 4356.7 188364   0.0000 0.0023  Dumpvalue::DumpElem


First I simply removed the overload operator "" (as_string). I still get the
same results and processing time is reduced drastically.
I don't really know why this way of stringifying (complex) perl types is
used here. 
Is there a special reason for it? Can't we simply use Data::Dumper (which
prints to a $var by default)?

[davebowman at hal2000 ]$ dprofpp
Total Elapsed Time = 35.85114 Seconds
   User+System Time = 33.54114 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  64.5   21.64 21.646 463568   0.0000 0.0000  File::Spec::Unix::canonpath
  22.9   7.698 31.602 231569   0.0000 0.0000  File::Spec::Unix::catfile
  19.7   6.607  6.607   2144   0.0031 0.0031
MOSES::MOBY::Cache::Registries::init
  18.6   6.251 37.772   2143   0.0029 0.0176
MOSES::MOBY::Generators::Utils::find_file
  6.80   2.282 13.448 231569   0.0000 0.0000  File::Spec::Unix::catdir

A lot of calls are made to MOSES::MOBY::Generators::Utils::find_file,
which in it's turn does a lot of calls with File::Spec. So I simply cache
the results.
Time is reduced to 17 seconds:

Total Elapsed Time = 17.34032 Seconds
   User+System Time = 13.20032 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  51.5   6.807  6.807   2144   0.0032 0.0032
MOSES::MOBY::Cache::Registries::init
  9.01   1.190  2.005 162858   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
  4.65   0.614  0.614 310751   0.0000 0.0000  XML::LibXML::Node::nodeName
  4.64   0.612  1.655  32171   0.0000 0.0001
XML::LibXML::Element::getChildrenByTagName
  4.48   0.592  0.592   5515   0.0001 0.0001
MOSES::MOBY::Cache::Central::_clean

More then half of the time still goes to building up the MOBY environment
for each processed element in the XML file. Isn't this a bit overkill? ;) Is
there a way to make a MOSES::MOBY factory that can handle the building up of
the moby environment just once in the same (mod_perl) call? Or is there a
reason this has been avoided?

I've included the patches for the result caching in
MOSES::MOBY::Generators::Utils::find_file . there's not much to it. My
version of MOSES::MOBY still is the one uploaded to cpan: Module MOSES::MOBY
(E/EK/EKAWAS/MOSES-MOBY-0.86.tar.gz)

happy hacking,
Kenny


On Tue, 4 Nov 2008, Kenny Billiau wrote:

> Hi,
>
> I've run a little testprogram to see where most of the time is spent. 
> I use a stripped down version of the xml file I've sent you. All but a 
> few ElementContainers and no sequence. Just to keep the processing 
> time low enough to debug.
>
>     use MOSES::MOBY::Parser;
>
>       open(FILE, '<', $ARGV[0]);
>       my @lines = <FILE>;
>       close FILE;
>
>       my $parser = MOSES::MOBY::Parser->new();
>       print $parser->parse( method => 'string', data => join(q{}, 
> @lines) );
>
>
> [davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl 
> mobysav_small.xml [davebowman at hal2000 ~]$ dprofpp 
> Exporter::Heavy::heavy_export has 14 unstacked calls in outer 
> Config::Simple::import_from has 1 unstacked calls in outer 
> utf8::AUTOLOAD has -1 unstacked calls in outer
> Config::Simple::  verbose has 1 unstacked calls in outer
> Config::Simple::  vars has 1 unstacked calls in outer 
> AutoLoader::AUTOLOAD has -3 unstacked calls in outer
> Fcntl::  AUTOLOAD has -6 unstacked calls in outer
> Fcntl: :__ANON__ has 6 unstacked calls in outer Exporter::export has 
> -14 unstacked calls in outer utf8::SWASHNEW has 1 unstacked calls in 
> outer Total Elapsed Time = 8.470220 Seconds  User+System Time = 
> 7.750220 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call 
> Csec/c  Name
>   68.1   5.280  5.280  69736   0.0001 0.0001  IO::Scalar::print
>   6.94   0.538  0.783  64088   0.0000 0.0000  Dumpvalue::stringify
>   6.04   0.468  7.173  33755   0.0000 0.0002  Dumpvalue::unwrap
>   4.30   0.333  0.333  47091   0.0000 0.0000  File::Spec::Unix::canonpath
>   2.43   0.188  7.157  33006   0.0000 0.0002  Dumpvalue::DumpElem
>   2.13   0.165  0.165  59776   0.0000 0.0000  overload::mycan
>   1.78   0.138  0.224  15990   0.0000 0.0000  MOSES::MOBY::Base::__ANON__
>   1.42   0.110  0.708    539   0.0002 0.0013
>   MOSES::MOBY::Cache::Central::_createDataTypeFromXML
>   1.39   0.108  0.108 142023   0.0000 0.0000  UNIVERSAL::isa
>   1.29   0.100  0.100    208   0.0005 0.0005
>   MOSES::MOBY::Cache::Registries::init
>   1.20   0.093  0.357  17359   0.0000 0.0000
overload::OverloadedStringify
>   1.16   0.090  0.090  30407   0.0000 0.0000  XML::LibXML::Node::nodeName
>   1.02   0.079  0.079    539   0.0001 0.0001
>   MOSES::MOBY::Cache::Central::_clean
>   0.99   0.077  0.418    207   0.0004 0.0020
>   MOSES::MOBY::Generators::Utils::find_file
>   0.94   0.073  0.203   3147   0.0000 0.0001
>   XML::LibXML::Element::getChildrenByTagName
>
> It's fairly obivous now ;)
>
> If I remove the overload operator in MOSES::MOBY::Base, time is 
> reduced 15fold! But for the 250KB file I've sent, it still takes us a 
> minute to parse.
>
> If I simply parse the file with XML::LibXML and no callbacks or custom 
> handlers, it's parsed instantly:
>
> [davebowman at hal2000 ~]$ /usr/bin/perl -d:DProf mobytest.pl 
> mobysav_small.xml
> [kebil at psbdev01 ~]$ dprofpp
> Total Elapsed Time = 0.079034 Seconds
>    User+System Time = 0.079034 Seconds
>
>
> -Kenny, slowly getting to the bottom of this ;)
>
>
> On Thu, 30 Oct 2008, Edward Kawas wrote:
>
>>   Now I can confirm that it is slow parsing and not an infinite loop! The
>>   XML
>>   file took over an hour on my virtual machine to parse. I will see what
I
>>   can
>>   do with other SAX parsers for Perl.
>>
>>   Thanks,
>>
>>   Eddie
>>
>>   -----Original Message-----
>>   From: moby-dev-bounces at lists.open-bio.org
>>   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Jos? Mar?a
>>   Fern?ndez Gonz?lez
>>   Sent: October-30-08 12:55 PM
>>   To: Core developer announcements
>>   Subject: Re: [MOBY-dev] XML parsing takes forever
>>
>>   Hi everybody,
>>   	I'm using the XML::LibXML SAX parser for other projects and tasks,
>>   and it can
>>   be really, really slow. You don't realize that until you are parsing a
>>   large
>>   file from a pipe, and you see that most of the time the program which
is
>>   parsing is CPU (and not I/O) bounded.
>>
>>    Best Regards,
>>     Jos? Mar?a
>>
>>   Edward Kawas wrote:
>> >   I think that you are right about the parser not going into an 
>> > infinite
>>   loop.
>> >   For some reason, the SAX parser seems to be really slow (although, I 
>> >   don't
>> >   want to say that until I know for sure).
>> > 
>> >   Eddie
>> > 
>> >   -----Original Message-----
>> >   From: moby-dev-bounces at lists.open-bio.org
>> >   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny
Billiau
>> >   Sent: October-30-08 11:56 AM
>> >   To: Core developer announcements
>> >   Subject: Re: [MOBY-dev] XML parsing takes forever
>> > 
>> >   Hi,
>> > 
>> >   thx for helping to investigate the problem!
>> > 
>> >   I'm pretty sure the parser gets out of it's mess. When you truncate
the
>> >   XML file by removing all but a few ElementContainers, it takes
awhile, 
>> >   but
>> 
>> >   it actually returns without error.
>> > 
>> >   And memory consuption doesn't seem to be a problem either, as mostly 
>> >   'top'
>> 
>> >   reveals that the invoked webservice script uses a steady amount of 
>> >   memory.
>> > 
>> >   thx,
>> >   Kenny
>> > 
>> >   On Thu, 30 Oct 2008, Edward Kawas wrote:
>> > 
>> > >   So this one might take some time... from what I can see, our parser

>> > >   goes
>> > >   into an infinite loop (although, I am not certain about that ...).
>> > > 
>> > >   I will play around with this XML file and get back to you.
>> > > 
>> > >   Thanks,
>> > > 
>> > >   Eddie
>> > > 
>> > >   -----Original Message-----
>> > >   From: moby-dev-bounces at lists.open-bio.org
>> > >   [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Kenny 
>> > >   Billiau
>> > >   Sent: October-30-08 8:03 AM
>> > >   To: Core developer announcements
>> > >   Subject: Re: [MOBY-dev] XML parsing takes forever
>> > > 
>> > >   Hi,
>> > > 
>> > >   here you go,
>> > > 
>> > >   Kenny
>> > > 
>> > >   On Thu, 30 Oct 2008, Edward Kawas wrote:
>> > > 
>> > > >   Hi,
>> > > > 
>> > > >   This is the right list! Could you send me the XML for your 
>> > > > second
>> >   service?
>> > > >   Thanks,
>> > > > 
>> > > >   Eddie
>> > >  _______________________________________________
>> > >   MOBY-dev mailing list
>> > >   MOBY-dev at lists.open-bio.org
>> > >   http://lists.open-bio.org/mailman/listinfo/moby-dev
>> > > 
>> >
>> 
>> 
>
>

-- 




From groscurt at mpiz-koeln.mpg.de  Fri Nov  7 08:56:53 2008
From: groscurt at mpiz-koeln.mpg.de (groscurt)
Date: Fri, 7 Nov 2008 09:56:53 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49131B32.2010605@pingoured.fr>
Message-ID: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>

Mhm i missed the intial email to that respond.

so i'm wondering - what is different from your list to the list from the one
from mobycentral http://moby.ucalgary.ca/moby/ValidateService ? 

And to the discussion about "working proper" services - it might be in the
document you will see, Sergio, but i'm really curios about the definition of
the phrase "work properly". So when does a service work properly ? 

of course, in my point of view, if it fulfills its registrated defintion -
so the question is how to test that ? 

I guess i wait for the document sergio was talking about ;-)

Cheers
andreas

----------------original message-----------------
From: "Pierre-Yves Chibon" 
To: "Core developer announcements" moby-dev at lists.open-bio.org
Date: Thu, 06 Nov 2008 17:28:34 +0100
-------------------------------------------------
> Sebastien Carrere wrote:
>> Here is my list of Dead services (Ihave got also list for the 3 past 
>> days for testRegistry and Inab registry) :
>>
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral
> 
> 
> Just by curiosity :
>   - do you have a list of working services ?
>   - do you have available your results from the testing repository ? (by 
> the way, why are you testing it ?)
> 
> Thanks in advance,
> 
> Regards,
> 
> Pierre
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
> 





From Sebastien.Carrere at toulouse.inra.fr  Fri Nov  7 09:28:32 2008
From: Sebastien.Carrere at toulouse.inra.fr (Sebastien Carrere)
Date: Fri, 07 Nov 2008 10:28:32 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
Message-ID: <49140A40.5070102@toulouse.inra.fr>

Hi,

my list contains the DeadServices provided on biomoby.org but also 
services where the following Perl test fails:

eval {$service_instance->execute(XMLinputlist =>[[]])};

The most frequent error is "Service execution failed: 500 Can't connect 
to localhost:8080 (connect: Connection refused)"

I don't know what is exactly done by 
http://moby.ucalgary.ca/moby/ValidateService .

Nevertheless, if we want to test if a webservice "works properly", we 
should first ask service developpers to register with test data.
This is already what we do using the PlayMoby framework.
By this way we survey our services ('not tested', 'ok', 'down') and 
generate a daily report.


Sebastien




groscurt wrote:
> Mhm i missed the intial email to that respond.
>
> so i'm wondering - what is different from your list to the list from the one
> from mobycentral http://moby.ucalgary.ca/moby/ValidateService ? 
>
> And to the discussion about "working proper" services - it might be in the
> document you will see, Sergio, but i'm really curios about the definition of
> the phrase "work properly". So when does a service work properly ? 
>
> of course, in my point of view, if it fulfills its registrated defintion -
> so the question is how to test that ? 
>
> I guess i wait for the document sergio was talking about ;-)
>
> Cheers
> andreas
>
> ----------------original message-----------------
> From: "Pierre-Yves Chibon" 
> To: "Core developer announcements" moby-dev at lists.open-bio.org
> Date: Thu, 06 Nov 2008 17:28:34 +0100
> -------------------------------------------------
>   
>> Sebastien Carrere wrote:
>>     
>>> Here is my list of Dead services (Ihave got also list for the 3 past 
>>> days for testRegistry and Inab registry) :
>>>
>>>       
> http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral
>   
>> Just by curiosity :
>>   - do you have a list of working services ?
>>   - do you have available your results from the testing repository ? (by 
>> the way, why are you testing it ?)
>>
>> Thanks in advance,
>>
>> Regards,
>>
>> Pierre
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>>     
>
>
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>   

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Sebastien_Carrere.vcf
Type: text/x-vcard
Size: 387 bytes
Desc: not available
URL: <http://lists.open-bio.org/pipermail/moby-dev/attachments/20081107/9e7a529d/attachment-0002.vcf>

From groscurt at mpiz-koeln.mpg.de  Fri Nov  7 10:20:11 2008
From: groscurt at mpiz-koeln.mpg.de (groscurt)
Date: Fri, 7 Nov 2008 11:20:11 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49140A40.5070102@toulouse.inra.fr>
Message-ID: <6371f6e1ab64fc3fdd519aeb0378eaa7@mpizmail.mpiz-koeln.mpg.de>

> Nevertheless, if we want to test if a webservice "works properly", we 
> should first ask service developpers to register with test data.
if i remember correctly, this was something we discussed in tokyo. 
I think we should consider to force people to give an example input for the
service, as this is nice a) for testing and b) to understand what the
service does.

But i remember one person (sorry I'm horrible with names) gave a good point
about service which need huge compute power and run for a longer time, which
then would bring them to their knees only because someone wants to test the
"integrity" of the service.

So service providers should be able to say whether those testing shall be
done for their service(s) or not, knowing that their services might be
highlighted as "not tested", but no "not working properly".

andreas

PS: It would be even nicer to force people more. E.g. if someone offers a
blast or any other algorithm to provide the information about what version
of the algorithm the service use or which database (in case of blast for
example) etc... but this can be become complicated.





From srramirez at uma.es  Fri Nov  7 14:11:30 2008
From: srramirez at uma.es (Sergio Ramirez Ramirez)
Date: Fri, 07 Nov 2008 15:11:30 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49140A40.5070102@toulouse.inra.fr>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr>
Message-ID: <49144C92.9020903@uma.es>

Hello all,

The only difference of the proposal with the list 
"http://moby.ucalgary.ca/moby/ValidateService"  is to include the 
information of this list in the service repository so it can be accessed 
using the API of MOBY.
It also provides the advantage of filter those services that haven't 
been validate when the user ask for a list of services.

When the service hasn't and example we only can test is the service is 
answering or not, so that is what means "work properly". In the INB we 
have examples associated with the services so is possible to test the 
services with more detail.

For non-sample services the status could be: Online, Timeout is the 
service is not answering and BadName when the service name is wrong. 
When an example was available is possible to check if the example is 
correct and the service doesn't returns with errors;  if the output type 
is the expected and/or if the output itself is correct using some 
mechanism to compare it with the one registered as example.

This is the new proposal in big details, but the main idea is to have 
the information that is now in the list available directly for clients.

Best regards,
Sergio.


Sebastien Carrere wrote:
> Hi,
>
> my list contains the DeadServices provided on biomoby.org but also 
> services where the following Perl test fails:
>
> eval {$service_instance->execute(XMLinputlist =>[[]])};
>
> The most frequent error is "Service execution failed: 500 Can't 
> connect to localhost:8080 (connect: Connection refused)"
>
> I don't know what is exactly done by 
> http://moby.ucalgary.ca/moby/ValidateService .
>
> Nevertheless, if we want to test if a webservice "works properly", we 
> should first ask service developpers to register with test data.
> This is already what we do using the PlayMoby framework.
> By this way we survey our services ('not tested', 'ok', 'down') and 
> generate a daily report.
>
>
> Sebastien
>
>
>
>
> groscurt wrote:
>> Mhm i missed the intial email to that respond.
>>
>> so i'm wondering - what is different from your list to the list from 
>> the one
>> from mobycentral http://moby.ucalgary.ca/moby/ValidateService ?
>> And to the discussion about "working proper" services - it might be 
>> in the
>> document you will see, Sergio, but i'm really curios about the 
>> definition of
>> the phrase "work properly". So when does a service work properly ?
>> of course, in my point of view, if it fulfills its registrated 
>> defintion -
>> so the question is how to test that ?
>> I guess i wait for the document sergio was talking about ;-)
>>
>> Cheers
>> andreas
>>
>> ----------------original message-----------------
>> From: "Pierre-Yves Chibon" To: "Core developer announcements" 
>> moby-dev at lists.open-bio.org
>> Date: Thu, 06 Nov 2008 17:28:34 +0100
>> -------------------------------------------------
>>  
>>> Sebastien Carrere wrote:
>>>    
>>>> Here is my list of Dead services (Ihave got also list for the 3 
>>>> past days for testRegistry and Inab registry) :
>>>>
>>>>       
>> http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral 
>>
>>  
>>> Just by curiosity :
>>>   - do you have a list of working services ?
>>>   - do you have available your results from the testing repository ? 
>>> (by the way, why are you testing it ?)
>>>
>>> Thanks in advance,
>>>
>>> Regards,
>>>
>>> Pierre
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>
>>>     
>>
>>
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>   
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>   


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 



From edward.kawas at gmail.com  Fri Nov  7 14:39:42 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Fri, 7 Nov 2008 06:39:42 -0800
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49144C92.9020903@uma.es>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>	<49140A40.5070102@toulouse.inra.fr>
	<49144C92.9020903@uma.es>
Message-ID: <4914533d.1f538c0a.541e.5585@mx.google.com>

Hello,

I wanted to chime in with my 2 cents!

Back in February during the Biohackathon, we (Canadian/Spanish/German
groups) came up with some predicates that we could use to add unit testing
information to a service[1]. 

Currently, JAVA can support parsing the RDF to get this information as well
as comparing the unit test information with the output for the service. I
have yet to add Perl support.

In addition, I have been working on a Dashboard panel that will help users
fill in unit test information for a particular service. In addition to
filling in the information, a service user can test the selected service and
save the unit test (in the service RDF).

Is the current proposal for testing services based on what we came up with
in February?

Thanks,

Eddie

[1] http://lists.open-bio.org/pipermail/moby-dev/2008-June/005061.html


-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Sergio Ramirez
Ramirez
Sent: November-07-08 6:12 AM
To: Core developer announcements
Subject: Re: [MOBY-dev] Dead services and how to figure them out

Hello all,

The only difference of the proposal with the list 
"http://moby.ucalgary.ca/moby/ValidateService"  is to include the 
information of this list in the service repository so it can be accessed 
using the API of MOBY.
It also provides the advantage of filter those services that haven't 
been validate when the user ask for a list of services.

When the service hasn't and example we only can test is the service is 
answering or not, so that is what means "work properly". In the INB we 
have examples associated with the services so is possible to test the 
services with more detail.

For non-sample services the status could be: Online, Timeout is the 
service is not answering and BadName when the service name is wrong. 
When an example was available is possible to check if the example is 
correct and the service doesn't returns with errors;  if the output type 
is the expected and/or if the output itself is correct using some 
mechanism to compare it with the one registered as example.

This is the new proposal in big details, but the main idea is to have 
the information that is now in the list available directly for clients.

Best regards,
Sergio.


Sebastien Carrere wrote:
> Hi,
>
> my list contains the DeadServices provided on biomoby.org but also 
> services where the following Perl test fails:
>
> eval {$service_instance->execute(XMLinputlist =>[[]])};
>
> The most frequent error is "Service execution failed: 500 Can't 
> connect to localhost:8080 (connect: Connection refused)"
>
> I don't know what is exactly done by 
> http://moby.ucalgary.ca/moby/ValidateService .
>
> Nevertheless, if we want to test if a webservice "works properly", we 
> should first ask service developpers to register with test data.
> This is already what we do using the PlayMoby framework.
> By this way we survey our services ('not tested', 'ok', 'down') and 
> generate a daily report.
>
>
> Sebastien
>
>
>
>
> groscurt wrote:
>> Mhm i missed the intial email to that respond.
>>
>> so i'm wondering - what is different from your list to the list from 
>> the one
>> from mobycentral http://moby.ucalgary.ca/moby/ValidateService ?
>> And to the discussion about "working proper" services - it might be 
>> in the
>> document you will see, Sergio, but i'm really curios about the 
>> definition of
>> the phrase "work properly". So when does a service work properly ?
>> of course, in my point of view, if it fulfills its registrated 
>> defintion -
>> so the question is how to test that ?
>> I guess i wait for the document sergio was talking about ;-)
>>
>> Cheers
>> andreas
>>
>> ----------------original message-----------------
>> From: "Pierre-Yves Chibon" To: "Core developer announcements" 
>> moby-dev at lists.open-bio.org
>> Date: Thu, 06 Nov 2008 17:28:34 +0100
>> -------------------------------------------------
>>  
>>> Sebastien Carrere wrote:
>>>    
>>>> Here is my list of Dead services (Ihave got also list for the 3 
>>>> past days for testRegistry and Inab registry) :
>>>>
>>>>       
>>
http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral 
>>
>>  
>>> Just by curiosity :
>>>   - do you have a list of working services ?
>>>   - do you have available your results from the testing repository ? 
>>> (by the way, why are you testing it ?)
>>>
>>> Thanks in advance,
>>>
>>> Regards,
>>>
>>> Pierre
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>
>>>     
>>
>>
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>   
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>   


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 

_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev




From srramirez at uma.es  Mon Nov 10 16:54:32 2008
From: srramirez at uma.es (Sergio Ramirez Ramirez)
Date: Mon, 10 Nov 2008 17:54:32 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <4914533d.1f538c0a.541e.5585@mx.google.com>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>	<49140A40.5070102@toulouse.inra.fr>	<49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com>
Message-ID: <49186748.5090402@uma.es>

Hi Eddie,

The idea is to define a list of status for the services, direct 
accessible from the list of them, that represents the results of the 
tests make over it.
If there aren't samples available, the only test that can be done is a 
ping to see is the service responds. In this case the status proposed are:

    * online: The service responds properly to the ping 
    * Timeout: If the time limit is got when trying to connect to the
      service.
    * NotFound: The service url is not found in the server
    * InternalServiceError: The service responds with an error related
      with the implementation of the service.
    * Disconnected: The connection with the service couldn't be established.

In addition, the NonAvailable status represents those services that have 
been down manually by the service provider for performing administrative 
tasks; and the unknown status when cannot be determine the status of the 
service.

When an example was available (defined in the RDF or in the extended 
registry of the INB, etc) can be make more specific test to the service. 
In this case, the set that we proposed is:

    * Tested: The service has been tested with the inputs given and no
      errors have been reported.
    * BadSample: The service respond with errors related with the input
      given.
    * NoSample: There aren't example for testing the service.

The output can be also used for test the behavior of the service. In 
this case we discussed two ways for testing:

    * TypeCorrect: The type of the data returned is the same or
      compatible with the expected one.
    * TypeIncorrect: If the data type is not compatible with the
      expected one.

And if  a way is provided for check the output content (regular 
expression or xpath) them there are two more status to contemplate:

    * OutputTested: The output content is check and fit with the results
      expected.
    * OutputFailed: in other case.

As you can see Eddie, the proposal is compatible with the one you 
mention. The only difference is the way to query the status of the 
service, that know can  be accessible using the functions of the MOBY 
Central..

Edward Kawas wrote:
> Hello,
>
> I wanted to chime in with my 2 cents!
>
> Back in February during the Biohackathon, we (Canadian/Spanish/German
> groups) came up with some predicates that we could use to add unit testing
> information to a service[1]. 
>
> Currently, JAVA can support parsing the RDF to get this information as well
> as comparing the unit test information with the output for the service. I
> have yet to add Perl support.
>
> In addition, I have been working on a Dashboard panel that will help users
> fill in unit test information for a particular service. In addition to
> filling in the information, a service user can test the selected service and
> save the unit test (in the service RDF).
>
> Is the current proposal for testing services based on what we came up with
> in February?
>
> Thanks,
>
> Eddie
>
> [1] http://lists.open-bio.org/pipermail/moby-dev/2008-June/005061.html
>
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Sergio Ramirez
> Ramirez
> Sent: November-07-08 6:12 AM
> To: Core developer announcements
> Subject: Re: [MOBY-dev] Dead services and how to figure them out
>
> Hello all,
>
> The only difference of the proposal with the list 
> "http://moby.ucalgary.ca/moby/ValidateService"  is to include the 
> information of this list in the service repository so it can be accessed 
> using the API of MOBY.
> It also provides the advantage of filter those services that haven't 
> been validate when the user ask for a list of services.
>
> When the service hasn't and example we only can test is the service is 
> answering or not, so that is what means "work properly". In the INB we 
> have examples associated with the services so is possible to test the 
> services with more detail.
>
> For non-sample services the status could be: Online, Timeout is the 
> service is not answering and BadName when the service name is wrong. 
> When an example was available is possible to check if the example is 
> correct and the service doesn't returns with errors;  if the output type 
> is the expected and/or if the output itself is correct using some 
> mechanism to compare it with the one registered as example.
>
> This is the new proposal in big details, but the main idea is to have 
> the information that is now in the list available directly for clients.
>
> Best regards,
> Sergio.
>
>
> Sebastien Carrere wrote:
>   
>> Hi,
>>
>> my list contains the DeadServices provided on biomoby.org but also 
>> services where the following Perl test fails:
>>
>> eval {$service_instance->execute(XMLinputlist =>[[]])};
>>
>> The most frequent error is "Service execution failed: 500 Can't 
>> connect to localhost:8080 (connect: Connection refused)"
>>
>> I don't know what is exactly done by 
>> http://moby.ucalgary.ca/moby/ValidateService .
>>
>> Nevertheless, if we want to test if a webservice "works properly", we 
>> should first ask service developpers to register with test data.
>> This is already what we do using the PlayMoby framework.
>> By this way we survey our services ('not tested', 'ok', 'down') and 
>> generate a daily report.
>>
>>
>> Sebastien
>>
>>
>>
>>
>> groscurt wrote:
>>     
>>> Mhm i missed the intial email to that respond.
>>>
>>> so i'm wondering - what is different from your list to the list from 
>>> the one
>>> from mobycentral http://moby.ucalgary.ca/moby/ValidateService ?
>>> And to the discussion about "working proper" services - it might be 
>>> in the
>>> document you will see, Sergio, but i'm really curios about the 
>>> definition of
>>> the phrase "work properly". So when does a service work properly ?
>>> of course, in my point of view, if it fulfills its registrated 
>>> defintion -
>>> so the question is how to test that ?
>>> I guess i wait for the document sergio was talking about ;-)
>>>
>>> Cheers
>>> andreas
>>>
>>> ----------------original message-----------------
>>> From: "Pierre-Yves Chibon" To: "Core developer announcements" 
>>> moby-dev at lists.open-bio.org
>>> Date: Thu, 06 Nov 2008 17:28:34 +0100
>>> -------------------------------------------------
>>>  
>>>       
>>>> Sebastien Carrere wrote:
>>>>    
>>>>         
>>>>> Here is my list of Dead services (Ihave got also list for the 3 
>>>>> past days for testRegistry and Inab registry) :
>>>>>
>>>>>       
>>>>>           
> http://lipm-bioinfo.toulouse.inra.fr/remora/sessions/blacklist.mobycentral 
>   
>>>  
>>>       
>>>> Just by curiosity :
>>>>   - do you have a list of working services ?
>>>>   - do you have available your results from the testing repository ? 
>>>> (by the way, why are you testing it ?)
>>>>
>>>> Thanks in advance,
>>>>
>>>> Regards,
>>>>
>>>> Pierre
>>>> _______________________________________________
>>>> MOBY-dev mailing list
>>>> MOBY-dev at lists.open-bio.org
>>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>>
>>>>     
>>>>         
>>>
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>   
>>>       
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>   
>>     
>
>
>   


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 



From markw at illuminae.com  Mon Nov 10 17:50:24 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Mon, 10 Nov 2008 09:50:24 -0800
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49186748.5090402@uma.es>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr> <49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com> <49186748.5090402@uma.es>
Message-ID: <op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>

On Mon, 10 Nov 2008 08:54:32 -0800, Sergio Ramirez Ramirez  
<srramirez at uma.es> wrote:


>   * Timeout: If the time limit is got when trying to connect to the
>       service.
>     * Disconnected: The connection with the service couldn't be  
> established.

What's the difference between these two?



> As you can see Eddie, the proposal is compatible with the one you  
> mention. The only difference is the way to query the status of the  
> service, that know can  be accessible using the functions of the MOBY  
> Central..


Thanks Sergio!  As usual, the INB has put together a very clear and  
thought-out proposal, and it is perfectly consistent with what we had in  
mind!  fantastic!

If I understand correctly, you currently keep this kind of metadata in the  
INB extended MOBY Central, right?  How tightly integrated is it with MOBY  
Central?  (i.e. have you changed the MOBY Central API such that you query  
for this metadata inside of the functions defined by the 'official' API,  
or is this data accessible via new API functions?)

We (my lab) are similarly interested in QoS issues, and where to store  
this kind of QoSD metadata.  We're looking at repositories such as  
BioCatalogue as a possibile store, since we don't think that this is the  
kind of metadata that the registry itself should be in control of (in  
fact, the registry may not even be capable of knowing!)

Do you or your colleagues at INB have strong opinions about where to store  
this metadata, and what the API should be to query for it?

best wishes!

Mark



From srramirez at uma.es  Mon Nov 10 18:37:25 2008
From: srramirez at uma.es (Sergio Ramirez Ramirez)
Date: Mon, 10 Nov 2008 19:37:25 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>	<49140A40.5070102@toulouse.inra.fr>
	<49144C92.9020903@uma.es>	<4914533d.1f538c0a.541e.5585@mx.google.com>
	<49186748.5090402@uma.es>
	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
Message-ID: <49187F65.7040502@uma.es>

Thanks Mark for your answer.

I respond "between lines":

Mark Wilkinson wrote:
> On Mon, 10 Nov 2008 08:54:32 -0800, Sergio Ramirez Ramirez 
> <srramirez at uma.es> wrote:
>
>
>>   * Timeout: If the time limit is got when trying to connect to the
>>       service.
>>     * Disconnected: The connection with the service couldn't be 
>> established.
>
> What's the difference between these two?
>
Really disconnected is more general than Timeout. It can be used when is 
not possible to connect with the service but not exactly due a timeout.
>
>
>> As you can see Eddie, the proposal is compatible with the one you 
>> mention. The only difference is the way to query the status of the 
>> service, that know can  be accessible using the functions of the MOBY 
>> Central..
>
>
> Thanks Sergio!  As usual, the INB has put together a very clear and 
> thought-out proposal, and it is perfectly consistent with what we had 
> in mind!  fantastic!
>
> If I understand correctly, you currently keep this kind of metadata in 
> the INB extended MOBY Central, right?  How tightly integrated is it 
> with MOBY Central?  (i.e. have you changed the MOBY Central API such 
> that you query for this metadata inside of the functions defined by 
> the 'official' API, or is this data accessible via new API functions?)
The idea is to make the information of the test done accessible thought 
the API. For that  we need to include the status and the last test done 
as new information of the registry. Related with the functions, the 
proposal was to extend the functions to recover the list of services so 
they returns also information about the status. Also was proposed that 
those services that don't pass the test weren't returned in the list by 
default so the users always got a list of working services
>
> We (my lab) are similarly interested in QoS issues, and where to store 
> this kind of QoSD metadata.  We're looking at repositories such as 
> BioCatalogue as a possibile store, since we don't think that this is 
> the kind of metadata that the registry itself should be in control of 
> (in fact, the registry may not even be capable of knowing!)
> Do you or your colleagues at INB have strong opinions about where to 
> store this metadata, and what the API should be to query for it?
In my personal opinion the information have to be stored in accesible 
way (web-service, xml or similar) so can be handle programmability. The 
real format and protocol used for stored them is not the most important 
thing.
>
> best wishes!
>
> Mark
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 



From markw at illuminae.com  Mon Nov 10 19:02:59 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Mon, 10 Nov 2008 11:02:59 -0800
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49187F65.7040502@uma.es>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr> <49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com> <49186748.5090402@uma.es>
	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
	<49187F65.7040502@uma.es>
Message-ID: <op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>

Hi Sergio!

it looks like we're back to the recurring question:  what metadata belongs  
IN the registry, and what metadata belongs elsewhere...

     <<sigh>>

I tend to have the opinion that only metadata necessary for discovery of a  
service should be in the registry itself, and that metadata about QoS,  
uptime, downtime, versioning, etc. all belongs somewhere else...  by  
"somewhere else" I mean (a) in the hands of the service provider  
themselves, or (b) in a third-party metadata repository.  This was one of  
the primary driving motivations behind our use of LSIDs in BioMoby, since  
the LSID allowed us to provide this non-registry metadata through a  
straightforward, predictable API that was independent of the registry  
API.  That way, we didn't have to modify the registry or the registry API  
every time we had a new kind of metadata!  I *still* believe this is the  
right choice... though I understand that sometimes efficiency of query is  
a compelling reason to have all of that metadata in the same place.   
Nevertheless, I don't think it should be a function of the registry to  
decide which services it will and will not show you!  The registry should  
simply tell you everything it knows, and it should be a client-side  
decision which of those services it displays to the end-user (IMO).  As  
such, I don't necessarily see the motivation for having this kind of  
metadata as part of the registry API...

How do others feel about this?  Is there *anyone* (other than Gbrowse  
Moby) who actually uses the LSID resolver at MOBY Central to retrieve this  
third-party metadata?  If not, is there a reason why not?

Martin, perhaps you can weigh-in on this question, as you had  
once-upon-a-time worked on the NetNanny project that would have monitored  
QoS... do you have a strong opinion one way or the other about where this  
metadata should live?  (LOL!  Asking Martin if he has a "strong opinion"  
is perhaps redundant ;-)  LOL!)

Best wishes all!

Mark








On Mon, 10 Nov 2008 10:37:25 -0800, Sergio Ramirez Ramirez  
<srramirez at uma.es> wrote:

> Thanks Mark for your answer.
>
> I respond "between lines":
>
> Mark Wilkinson wrote:
>> On Mon, 10 Nov 2008 08:54:32 -0800, Sergio Ramirez Ramirez  
>> <srramirez at uma.es> wrote:
>>
>>
>>>   * Timeout: If the time limit is got when trying to connect to the
>>>       service.
>>>     * Disconnected: The connection with the service couldn't be  
>>> established.
>>
>> What's the difference between these two?
>>
> Really disconnected is more general than Timeout. It can be used when is  
> not possible to connect with the service but not exactly due a timeout.
>>
>>
>>> As you can see Eddie, the proposal is compatible with the one you  
>>> mention. The only difference is the way to query the status of the  
>>> service, that know can  be accessible using the functions of the MOBY  
>>> Central..
>>
>>
>> Thanks Sergio!  As usual, the INB has put together a very clear and  
>> thought-out proposal, and it is perfectly consistent with what we had  
>> in mind!  fantastic!
>>
>> If I understand correctly, you currently keep this kind of metadata in  
>> the INB extended MOBY Central, right?  How tightly integrated is it  
>> with MOBY Central?  (i.e. have you changed the MOBY Central API such  
>> that you query for this metadata inside of the functions defined by the  
>> 'official' API, or is this data accessible via new API functions?)
> The idea is to make the information of the test done accessible thought  
> the API. For that  we need to include the status and the last test done  
> as new information of the registry. Related with the functions, the  
> proposal was to extend the functions to recover the list of services so  
> they returns also information about the status. Also was proposed that  
> those services that don't pass the test weren't returned in the list by  
> default so the users always got a list of working services
>>
>> We (my lab) are similarly interested in QoS issues, and where to store  
>> this kind of QoSD metadata.  We're looking at repositories such as  
>> BioCatalogue as a possibile store, since we don't think that this is  
>> the kind of metadata that the registry itself should be in control of  
>> (in fact, the registry may not even be capable of knowing!)
>> Do you or your colleagues at INB have strong opinions about where to  
>> store this metadata, and what the API should be to query for it?
> In my personal opinion the information have to be stored in accesible  
> way (web-service, xml or similar) so can be handle programmability. The  
> real format and protocol used for stored them is not the most important  
> thing.
>>
>> best wishes!
>>
>> Mark
>>
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>
>
>



-- 
Mark D Wilkinson, PI Bioinformatics
Assistant Professor, Medical Genetics
The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
Providence Heart + Lung Institute
University of British Columbia - St. Paul's Hospital
Vancouver, BC, Canada


From martin.senger at gmail.com  Mon Nov 10 19:13:57 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 11 Nov 2008 00:43:57 +0530
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr> <49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com> <49186748.5090402@uma.es>
	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
	<49187F65.7040502@uma.es>
	<op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
Message-ID: <4d93f07c0811101113md33d8adif4c20ed4a6144f29@mail.gmail.com>

> I tend to have the opinion that only metadata necessary for discovery of a
> service should be in the registry itself, and that metadata about QoS,
> uptime, downtime, versioning, etc. all belongs somewhere else...  by
> "somewhere else" I mean (a) in the hands of the service provider themselves,
> or (b) in a third-party metadata repository.


Exactly my opinion (strong or weak :-)). Let's not pollute the registry. The
"smart" clients should anyway make some caches of the metadata so the
concern about the performance is a weak one I believe.

Cheers,
Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger


From jmfernandez at cnio.es  Mon Nov 10 19:33:09 2008
From: jmfernandez at cnio.es (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Fern=E1ndez_Gonz=E1lez?=)
Date: Mon, 10 Nov 2008 20:33:09 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <4d93f07c0811101113md33d8adif4c20ed4a6144f29@mail.gmail.com>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>	<49140A40.5070102@toulouse.inra.fr>
	<49144C92.9020903@uma.es>	<4914533d.1f538c0a.541e.5585@mx.google.com>
	<49186748.5090402@uma.es>	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>	<49187F65.7040502@uma.es>	<op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
	<4d93f07c0811101113md33d8adif4c20ed4a6144f29@mail.gmail.com>
Message-ID: <49188C75.1060707@cnio.es>

But, what does it happen if a service provider suddenly "disappears" (e.g. DNS
failure)? If QoS is stored just besides the service host, the Central Registry
could not fetch the QoS information in that case.

And, what does it happen if a QoS service provider suddenly "disappears" (e.g.
again DNS failure)? Service is usable, service is reachable, but QoS isn't...

Also, QoS is perceived in a different way depending on the place where your
client is plugged. The real QoS of a japanese service is different when you
are in Japan than when you are in Europe.

So, I have just realized that perhaps a possible answer is having replicated
QoS metadata servers spread around the world, related in some way to MOBY Central.

	Just my two euro cents...
		Jos? Mar?a

Martin Senger wrote:
>> I tend to have the opinion that only metadata necessary for discovery of a
>> service should be in the registry itself, and that metadata about QoS,
>> uptime, downtime, versioning, etc. all belongs somewhere else...  by
>> "somewhere else" I mean (a) in the hands of the service provider themselves,
>> or (b) in a third-party metadata repository.
> 
> 
> Exactly my opinion (strong or weak :-)). Let's not pollute the registry. The
> "smart" clients should anyway make some caches of the metadata so the
> concern about the performance is a weak one I believe.
> 
> Cheers,
> Martin
> 

-- 
"There is no reason why anybody would want a computer in their home" -
	Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???

"Premature optimization is the root of all evil." - Donald Knuth

Jos? Mar?a Fern?ndez Gonz?lez
Tlfn: (+34) 91 732 80 00 / 91 224 69 00 (ext 3061)
e-mail: jmfernandez at cnio.es		Fax: (+34) 91 224 69 76
Unidad del Instituto Nacional de Bioinform?tica
Biolog?a Estructural y Biocomputaci?n	Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncol?gicas
C.P.: 28029				Zip Code: 28029
C/. Melchor Fern?ndez Almagro, 3	Madrid (Spain)

**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en su caso los ficheros adjuntos, pueden contener informaci?n protegida para el uso exclusivo de su destinatario. Se proh?be la distribuci?n, reproducci?n o cualquier otro tipo de transmisi?n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.



From martin.senger at gmail.com  Tue Nov 11 02:48:00 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Tue, 11 Nov 2008 08:18:00 +0530
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <49188C75.1060707@cnio.es>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr> <49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com> <49186748.5090402@uma.es>
	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
	<49187F65.7040502@uma.es>
	<op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
	<4d93f07c0811101113md33d8adif4c20ed4a6144f29@mail.gmail.com>
	<49188C75.1060707@cnio.es>
Message-ID: <4d93f07c0811101848v56734722j77c87c8a473f2550@mail.gmail.com>

> So, I have just realized that perhaps a possible answer is having
> replicated
> QoS metadata servers spread around the world


That's what I meant by "smart" clients caching it...

Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger


From schoof at mpiz-koeln.mpg.de  Tue Nov 11 08:05:42 2008
From: schoof at mpiz-koeln.mpg.de (Heiko Schoof)
Date: Tue, 11 Nov 2008 09:05:42 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>
	<49140A40.5070102@toulouse.inra.fr> <49144C92.9020903@uma.es>
	<4914533d.1f538c0a.541e.5585@mx.google.com>
	<49186748.5090402@uma.es>
	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>
	<49187F65.7040502@uma.es>
	<op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
Message-ID: <8C3FFF0A-07AD-43FD-8BD0-E75619193E13@mpiz-koeln.mpg.de>

Hi Mark and others,

I tend to have strong opinions, but utter them softly (meaning both  
"hard to hear" and "gently" ;-)

I strongly lean towards keeping QoS separate from the core registry,  
many arguments have been given. But colocalization would make sense.  
We have been discussing failure redundancy for Moby Central, and a  
mirror plus QoS on the same server would make sense. If a client has  
no access to Central data, it doesn't need QoS. On the other hand, the  
central Central used for registration i.e. writing to the registry  
doesn't need QoS. That allows us to keep provider-provided metadata  
(like "this is the authoritative service for x" or "this is a mirror/ 
backup service for x") separate from real, externally measured QoS.

I actually might have a student that could put QoS awareness into our  
Jabba aggregator, which would make a lot of sense.

Greetings from Cologne in pouring rain,

HEiko

On 10.11.2008, at 20:02, Mark Wilkinson wrote:

> Hi Sergio!
>
> it looks like we're back to the recurring question:  what metadata  
> belongs IN the registry, and what metadata belongs elsewhere...
>
>    <<sigh>>
>
> I tend to have the opinion that only metadata necessary for  
> discovery of a service should be in the registry itself, and that  
> metadata about QoS, uptime, downtime, versioning, etc. all belongs  
> somewhere else...  by "somewhere else" I mean (a) in the hands of  
> the service provider themselves, or (b) in a third-party metadata  
> repository.  This was one of the primary driving motivations behind  
> our use of LSIDs in BioMoby, since the LSID allowed us to provide  
> this non-registry metadata through a straightforward, predictable  
> API that was independent of the registry API.  That way, we didn't  
> have to modify the registry or the registry API every time we had a  
> new kind of metadata!  I *still* believe this is the right choice...  
> though I understand that sometimes efficiency of query is a  
> compelling reason to have all of that metadata in the same place.   
> Nevertheless, I don't think it should be a function of the registry  
> to decide which services it will and will not show you!  The  
> registry should simply tell you everything it knows, and it should  
> be a client-side decision which of those services it displays to the  
> end-user (IMO).  As such, I don't necessarily see the motivation for  
> having this kind of metadata as part of the registry API...
>
> How do others feel about this?  Is there *anyone* (other than  
> Gbrowse Moby) who actually uses the LSID resolver at MOBY Central to  
> retrieve this third-party metadata?  If not, is there a reason why  
> not?
>
> Martin, perhaps you can weigh-in on this question, as you had once- 
> upon-a-time worked on the NetNanny project that would have monitored  
> QoS... do you have a strong opinion one way or the other about where  
> this metadata should live?  (LOL!  Asking Martin if he has a "strong  
> opinion" is perhaps redundant ;-)  LOL!)
>
> Best wishes all!
>
> Mark
>
>
>
>
>
>
>
>
> On Mon, 10 Nov 2008 10:37:25 -0800, Sergio Ramirez Ramirez <srramirez at uma.es 
> > wrote:
>
>> Thanks Mark for your answer.
>>
>> I respond "between lines":
>>
>> Mark Wilkinson wrote:
>>> On Mon, 10 Nov 2008 08:54:32 -0800, Sergio Ramirez Ramirez <srramirez at uma.es 
>>> > wrote:
>>>
>>>
>>>>  * Timeout: If the time limit is got when trying to connect to the
>>>>      service.
>>>>    * Disconnected: The connection with the service couldn't be  
>>>> established.
>>>
>>> What's the difference between these two?
>>>
>> Really disconnected is more general than Timeout. It can be used  
>> when is not possible to connect with the service but not exactly  
>> due a timeout.
>>>
>>>
>>>> As you can see Eddie, the proposal is compatible with the one you  
>>>> mention. The only difference is the way to query the status of  
>>>> the service, that know can  be accessible using the functions of  
>>>> the MOBY Central..
>>>
>>>
>>> Thanks Sergio!  As usual, the INB has put together a very clear  
>>> and thought-out proposal, and it is perfectly consistent with what  
>>> we had in mind!  fantastic!
>>>
>>> If I understand correctly, you currently keep this kind of  
>>> metadata in the INB extended MOBY Central, right?  How tightly  
>>> integrated is it with MOBY Central?  (i.e. have you changed the  
>>> MOBY Central API such that you query for this metadata inside of  
>>> the functions defined by the 'official' API, or is this data  
>>> accessible via new API functions?)
>> The idea is to make the information of the test done accessible  
>> thought the API. For that  we need to include the status and the  
>> last test done as new information of the registry. Related with the  
>> functions, the proposal was to extend the functions to recover the  
>> list of services so they returns also information about the status.  
>> Also was proposed that those services that don't pass the test  
>> weren't returned in the list by default so the users always got a  
>> list of working services
>>>
>>> We (my lab) are similarly interested in QoS issues, and where to  
>>> store this kind of QoSD metadata.  We're looking at repositories  
>>> such as BioCatalogue as a possibile store, since we don't think  
>>> that this is the kind of metadata that the registry itself should  
>>> be in control of (in fact, the registry may not even be capable of  
>>> knowing!)
>>> Do you or your colleagues at INB have strong opinions about where  
>>> to store this metadata, and what the API should be to query for it?
>> In my personal opinion the information have to be stored in  
>> accesible way (web-service, xml or similar) so can be handle  
>> programmability. The real format and protocol used for stored them  
>> is not the most important thing.
>>>
>>> best wishes!
>>>
>>> Mark
>>>
>>> _______________________________________________
>>> MOBY-dev mailing list
>>> MOBY-dev at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>
>>
>>
>
>
>
> -- 
> Mark D Wilkinson, PI Bioinformatics
> Assistant Professor, Medical Genetics
> The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary  
> Research
> Providence Heart + Lung Institute
> University of British Columbia - St. Paul's Hospital
> Vancouver, BC, Canada
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev



From srramirez at uma.es  Tue Nov 11 09:53:53 2008
From: srramirez at uma.es (Sergio Ramirez Ramirez)
Date: Tue, 11 Nov 2008 10:53:53 +0100
Subject: [MOBY-dev] Dead services and how to figure them out
In-Reply-To: <8C3FFF0A-07AD-43FD-8BD0-E75619193E13@mpiz-koeln.mpg.de>
References: <43a1ad913338baa422edb1ea3cc49bee@mpizmail.mpiz-koeln.mpg.de>	<49140A40.5070102@toulouse.inra.fr>
	<49144C92.9020903@uma.es>	<4914533d.1f538c0a.541e.5585@mx.google.com>	<49186748.5090402@uma.es>	<op.ukezuaqgnbznux@dd0710001l.icapture.ubc.ca>	<49187F65.7040502@uma.es>	<op.uke269l2nbznux@dd0710001l.icapture.ubc.ca>
	<8C3FFF0A-07AD-43FD-8BD0-E75619193E13@mpiz-koeln.mpg.de>
Message-ID: <49195631.4090301@uma.es>

Hi Heiko, Mark and others

We can keep QoS separated from core registry as long as you can access 
automatically by clients, parsing RDF can be a possible solution.

Heiko Schoof wrote:
> Hi Mark and others,
>
> I tend to have strong opinions, but utter them softly (meaning both 
> "hard to hear" and "gently" ;-)
>
> I strongly lean towards keeping QoS separate from the core registry, 
> many arguments have been given. But colocalization would make sense. 
> We have been discussing failure redundancy for Moby Central, and a 
> mirror plus QoS on the same server would make sense. If a client has 
> no access to Central data, it doesn't need QoS. On the other hand, the 
> central Central used for registration i.e. writing to the registry 
> doesn't need QoS. That allows us to keep provider-provided metadata 
> (like "this is the authoritative service for x" or "this is a 
> mirror/backup service for x") separate from real, externally measured 
> QoS.
>
> I actually might have a student that could put QoS awareness into our 
> Jabba aggregator, which would make a lot of sense.
>
> Greetings from Cologne in pouring rain,
>
> HEiko
>
> On 10.11.2008, at 20:02, Mark Wilkinson wrote:
>
>> Hi Sergio!
>>
>> it looks like we're back to the recurring question:  what metadata 
>> belongs IN the registry, and what metadata belongs elsewhere...
>>
>>    <<sigh>>
>>
>> I tend to have the opinion that only metadata necessary for discovery 
>> of a service should be in the registry itself, and that metadata 
>> about QoS, uptime, downtime, versioning, etc. all belongs somewhere 
>> else...  by "somewhere else" I mean (a) in the hands of the service 
>> provider themselves, or (b) in a third-party metadata repository.  
>> This was one of the primary driving motivations behind our use of 
>> LSIDs in BioMoby, since the LSID allowed us to provide this 
>> non-registry metadata through a straightforward, predictable API that 
>> was independent of the registry API.  That way, we didn't have to 
>> modify the registry or the registry API every time we had a new kind 
>> of metadata!  I *still* believe this is the right choice... though I 
>> understand that sometimes efficiency of query is a compelling reason 
>> to have all of that metadata in the same place.  Nevertheless, I 
>> don't think it should be a function of the registry to decide which 
>> services it will and will not show you!  The registry should simply 
>> tell you everything it knows, and it should be a client-side decision 
>> which of those services it displays to the end-user (IMO).  As such, 
>> I don't necessarily see the motivation for having this kind of 
>> metadata as part of the registry API...
>>
>> How do others feel about this?  Is there *anyone* (other than Gbrowse 
>> Moby) who actually uses the LSID resolver at MOBY Central to retrieve 
>> this third-party metadata?  If not, is there a reason why not?
>>
>> Martin, perhaps you can weigh-in on this question, as you had 
>> once-upon-a-time worked on the NetNanny project that would have 
>> monitored QoS... do you have a strong opinion one way or the other 
>> about where this metadata should live?  (LOL!  Asking Martin if he 
>> has a "strong opinion" is perhaps redundant ;-)  LOL!)
>>
>> Best wishes all!
>>
>> Mark
>>
>>
>>
>>
>>
>>
>>
>>
>> On Mon, 10 Nov 2008 10:37:25 -0800, Sergio Ramirez Ramirez 
>> <srramirez at uma.es> wrote:
>>
>>> Thanks Mark for your answer.
>>>
>>> I respond "between lines":
>>>
>>> Mark Wilkinson wrote:
>>>> On Mon, 10 Nov 2008 08:54:32 -0800, Sergio Ramirez Ramirez 
>>>> <srramirez at uma.es> wrote:
>>>>
>>>>
>>>>>  * Timeout: If the time limit is got when trying to connect to the
>>>>>      service.
>>>>>    * Disconnected: The connection with the service couldn't be 
>>>>> established.
>>>>
>>>> What's the difference between these two?
>>>>
>>> Really disconnected is more general than Timeout. It can be used 
>>> when is not possible to connect with the service but not exactly due 
>>> a timeout.
>>>>
>>>>
>>>>> As you can see Eddie, the proposal is compatible with the one you 
>>>>> mention. The only difference is the way to query the status of the 
>>>>> service, that know can  be accessible using the functions of the 
>>>>> MOBY Central..
>>>>
>>>>
>>>> Thanks Sergio!  As usual, the INB has put together a very clear and 
>>>> thought-out proposal, and it is perfectly consistent with what we 
>>>> had in mind!  fantastic!
>>>>
>>>> If I understand correctly, you currently keep this kind of metadata 
>>>> in the INB extended MOBY Central, right?  How tightly integrated is 
>>>> it with MOBY Central?  (i.e. have you changed the MOBY Central API 
>>>> such that you query for this metadata inside of the functions 
>>>> defined by the 'official' API, or is this data accessible via new 
>>>> API functions?)
>>> The idea is to make the information of the test done accessible 
>>> thought the API. For that  we need to include the status and the 
>>> last test done as new information of the registry. Related with the 
>>> functions, the proposal was to extend the functions to recover the 
>>> list of services so they returns also information about the status. 
>>> Also was proposed that those services that don't pass the test 
>>> weren't returned in the list by default so the users always got a 
>>> list of working services
>>>>
>>>> We (my lab) are similarly interested in QoS issues, and where to 
>>>> store this kind of QoSD metadata.  We're looking at repositories 
>>>> such as BioCatalogue as a possibile store, since we don't think 
>>>> that this is the kind of metadata that the registry itself should 
>>>> be in control of (in fact, the registry may not even be capable of 
>>>> knowing!)
>>>> Do you or your colleagues at INB have strong opinions about where 
>>>> to store this metadata, and what the API should be to query for it?
>>> In my personal opinion the information have to be stored in 
>>> accesible way (web-service, xml or similar) so can be handle 
>>> programmability. The real format and protocol used for stored them 
>>> is not the most important thing.
>>>>
>>>> best wishes!
>>>>
>>>> Mark
>>>>
>>>> _______________________________________________
>>>> MOBY-dev mailing list
>>>> MOBY-dev at lists.open-bio.org
>>>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>>>>
>>>
>>>
>>
>>
>>
>> -- 
>> Mark D Wilkinson, PI Bioinformatics
>> Assistant Professor, Medical Genetics
>> The James Hogg iCAPTURE Centre for Cardiovascular and Pulmonary Research
>> Providence Heart + Lung Institute
>> University of British Columbia - St. Paul's Hospital
>> Vancouver, BC, Canada
>> _______________________________________________
>> MOBY-dev mailing list
>> MOBY-dev at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 



From groscurt at mpiz-koeln.mpg.de  Wed Nov 12 14:36:02 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 12 Nov 2008 15:36:02 +0100
Subject: [MOBY-dev] Compilation error in jmoby
Message-ID: <491AE9D2.9070804@mpiz-koeln.mpg.de>

Hi,

i just made an update of the jmoby and i get several errors concerning

Unhandled exception type NoSuccessException

in
moby/src/main/org/biomoby/client/GraphsServlet
moby/src/main/org/biomoby/service/dashboard/RegistryModel
moby/src/main/org/biomoby/service/generator/DataTypesGenerator
moby/src/main/org/biomoby/service/generator/ServicesGenerator

they apparently all coming from the getDataTypes method in the central 
classes.

Cheers
andreas

-- 
=====================================================
  Dipl. Bioinf. Andreas Groscurth
  Bioinformatics Software Developer
  Plant Computational Biology group
  Max-Planck Institute for plant breeding research
  Carl-von-Linne Weg 10
  50829 Cologne
  Germany
  +49(0) 221 5062449
=====================================================



From martin.senger at gmail.com  Wed Nov 12 14:48:48 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Wed, 12 Nov 2008 20:18:48 +0530
Subject: [MOBY-dev] Compilation error in jmoby
In-Reply-To: <491AE9D2.9070804@mpiz-koeln.mpg.de>
References: <491AE9D2.9070804@mpiz-koeln.mpg.de>
Message-ID: <4d93f07c0811120648x55a2d8bctc206a85ddd5afc3e@mail.gmail.com>

No, I cannot confirm the error. Everything works fine in my local copy. I
have 'cvs update -dP', and make 'ant clean compile' and no error reported,
compilation successful.

Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger


From edward.kawas at gmail.com  Wed Nov 12 14:49:53 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 12 Nov 2008 06:49:53 -0800
Subject: [MOBY-dev] Compilation error in jmoby
In-Reply-To: <491AE9D2.9070804@mpiz-koeln.mpg.de>
References: <491AE9D2.9070804@mpiz-koeln.mpg.de>
Message-ID: <491aed18.1f588c0a.2404.fffffcc7@mx.google.com>

I am not having any problems compiling.

What are the exact errors?

Eddie

-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Andreas Groscurth
Sent: November-12-08 6:36 AM
To: Core developer announcements
Subject: [MOBY-dev] Compilation error in jmoby

Hi,

i just made an update of the jmoby and i get several errors concerning

Unhandled exception type NoSuccessException

in
moby/src/main/org/biomoby/client/GraphsServlet
moby/src/main/org/biomoby/service/dashboard/RegistryModel
moby/src/main/org/biomoby/service/generator/DataTypesGenerator
moby/src/main/org/biomoby/service/generator/ServicesGenerator

they apparently all coming from the getDataTypes method in the central 
classes.

Cheers
andreas

-- 
=====================================================
  Dipl. Bioinf. Andreas Groscurth
  Bioinformatics Software Developer
  Plant Computational Biology group
  Max-Planck Institute for plant breeding research
  Carl-von-Linne Weg 10
  50829 Cologne
  Germany
  +49(0) 221 5062449
=====================================================

_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev



From groscurt at mpiz-koeln.mpg.de  Wed Nov 12 15:01:22 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 12 Nov 2008 16:01:22 +0100
Subject: [MOBY-dev] Compilation error in jmoby
In-Reply-To: <491aed18.1f588c0a.2404.fffffcc7@mx.google.com>
References: <491AE9D2.9070804@mpiz-koeln.mpg.de>
	<491aed18.1f588c0a.2404.fffffcc7@mx.google.com>
Message-ID: <491AEFC2.4060908@mpiz-koeln.mpg.de>

then i guess something with my eclipse is broken.

in the Central class there is the method "getDataTypes"  which throws an 
NoSuccessException.
All methods which call that do not catch that exception or throw them.

Thats at least my local problem here ;-)

Edward Kawas wrote:
> I am not having any problems compiling.
>
> What are the exact errors?
>
> Eddie
>
> -----Original Message-----
> From: moby-dev-bounces at lists.open-bio.org
> [mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Andreas Groscurth
> Sent: November-12-08 6:36 AM
> To: Core developer announcements
> Subject: [MOBY-dev] Compilation error in jmoby
>
> Hi,
>
> i just made an update of the jmoby and i get several errors concerning
>
> Unhandled exception type NoSuccessException
>
> in
> moby/src/main/org/biomoby/client/GraphsServlet
> moby/src/main/org/biomoby/service/dashboard/RegistryModel
> moby/src/main/org/biomoby/service/generator/DataTypesGenerator
> moby/src/main/org/biomoby/service/generator/ServicesGenerator
>
> they apparently all coming from the getDataTypes method in the central 
> classes.
>
> Cheers
> andreas
>
>   


-- 
=====================================================
  Dipl. Bioinf. Andreas Groscurth
  Bioinformatics Software Developer
  Plant Computational Biology group
  Max-Planck Institute for plant breeding research
  Carl-von-Linne Weg 10
  50829 Cologne
  Germany
  +49(0) 221 5062449
=====================================================



From edward.kawas at gmail.com  Wed Nov 12 20:06:59 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Wed, 12 Nov 2008 12:06:59 -0800
Subject: [MOBY-dev] moby unit testing
Message-ID: <491b376a.2a528c0a.1c17.15fa@mx.google.com>

Hi,

Mark and I have made available the new panel for, the webstart version of,
Dashboard (http://moby.ucalgary.ca/dashboard/) that can be used for creating
unit tests for Biomoby services. 

If you are at all interested, please try it out and let the list what you
think about it.

Thanks,

Eddie



From skhadar at gmail.com  Fri Nov 14 13:09:32 2008
From: skhadar at gmail.com (Shameer Khadar)
Date: Fri, 14 Nov 2008 18:39:32 +0530
Subject: [MOBY-dev] Reg. WebServices
Message-ID: <b6ff81950811140509o462e2c41rca21b0dfb9ccf447@mail.gmail.com>

Dear MOBY team,

I recently worked on an integrated bioinformatics  server. Currently the
servers is running on a single node CentOS box. Several users wanted to run
this using multiple queries, hence we would like to extend this as a Web
Service (SOAP/REST based), Can you guys tell me how can I deploy it using
MOBY. Is there any best practices guide to start development of webservices
?

Many thanks in advance,
-- 
K. Shameer
------------------------------------------------------------
Computational Biology Lab, NCBS - TIFR
Molecular Biophysics Unit, IISc


From srramirez at uma.es  Fri Nov 14 14:45:52 2008
From: srramirez at uma.es (Sergio Ramirez Ramirez)
Date: Fri, 14 Nov 2008 15:45:52 +0100
Subject: [MOBY-dev] Reg. WebServices
In-Reply-To: <b6ff81950811140509o462e2c41rca21b0dfb9ccf447@mail.gmail.com>
References: <b6ff81950811140509o462e2c41rca21b0dfb9ccf447@mail.gmail.com>
Message-ID: <491D8F20.4090403@uma.es>

Hello Shameer

In the INB we have some tutorials about how to deploy MOBY Services 
please take a look of this link 
(http://chirimoyo.ac.uma.es/wiki/index.php/MOWServ:Registration) and 
feel free of ask anything you don't understand.

Best Regards, Sergio


Shameer Khadar wrote:
> Dear MOBY team,
>
> I recently worked on an integrated bioinformatics  server. Currently the
> servers is running on a single node CentOS box. Several users wanted to run
> this using multiple queries, hence we would like to extend this as a Web
> Service (SOAP/REST based), Can you guys tell me how can I deploy it using
> MOBY. Is there any best practices guide to start development of webservices
> ?
>
> Many thanks in advance,
>   


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 



From edward.kawas at gmail.com  Fri Nov 14 18:20:16 2008
From: edward.kawas at gmail.com (Edward Kawas)
Date: Fri, 14 Nov 2008 10:20:16 -0800
Subject: [MOBY-dev] Reg. WebServices
In-Reply-To: <491D8F20.4090403@uma.es>
References: <b6ff81950811140509o462e2c41rca21b0dfb9ccf447@mail.gmail.com>
	<491D8F20.4090403@uma.es>
Message-ID: <491dc16e.16538c0a.2bc7.ffffb587@mx.google.com>

Hi,

If you are interested in how to go about developing services in perl:
http://search.cpan.org/dist/MOSES-MOBY/

Or Java:

http://biomoby.open-bio.org/CVS_CONTENT/moby-live/Java/docs/Moses.html

Hope this helps,

Eddie

-----Original Message-----
From: moby-dev-bounces at lists.open-bio.org
[mailto:moby-dev-bounces at lists.open-bio.org] On Behalf Of Sergio Ramirez
Ramirez
Sent: November-14-08 6:46 AM
To: Core developer announcements
Subject: Re: [MOBY-dev] Reg. WebServices

Hello Shameer

In the INB we have some tutorials about how to deploy MOBY Services 
please take a look of this link 
(http://chirimoyo.ac.uma.es/wiki/index.php/MOWServ:Registration) and 
feel free of ask anything you don't understand.

Best Regards, Sergio


Shameer Khadar wrote:
> Dear MOBY team,
>
> I recently worked on an integrated bioinformatics  server. Currently the
> servers is running on a single node CentOS box. Several users wanted to
run
> this using multiple queries, hence we would like to extend this as a Web
> Service (SOAP/REST based), Can you guys tell me how can I deploy it using
> MOBY. Is there any best practices guide to start development of
webservices
> ?
>
> Many thanks in advance,
>   


-- 
Sergio Ram?rez Ram?rez
Instituto Nacional de Bioinform?tica (INB)
Integrated Bioinformatics Node (GNV-5)
Dpto. de Arquitectura de Computadores
Campus Universitario de Teatinos, despacho 2.3.9a
29071 M?laga (Spain) +34 95 213 3387


"Short-term decisions tend to fail in the long-term."
            Frank Herbert, God Emperor of Dune 

_______________________________________________
MOBY-dev mailing list
MOBY-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/moby-dev




From groscurt at mpiz-koeln.mpg.de  Wed Nov 19 15:37:35 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 19 Nov 2008 16:37:35 +0100
Subject: [MOBY-dev] Dashboard as Webstart
Message-ID: <492432BF.8040809@mpiz-koeln.mpg.de>

Hi all,

as you might know I have created a Java Webstart version of the 
Dashboard which is available already on Biomoby.org.

I have now added some ant tasks in the dashboardBuild.xml file to enable 
the creation of the Dashboard Webstart for everyone and checked it into 
the cvs.

You can now create your own Dashboard Webstart by typing 'ant 
dashboard-webstart'.

You will be guided through this creation process and you also can find 
the documentation in docs/DashboardWebstart.html.

For any bugs, errors or comments (I'm not sure if the ant code is way 
too complex) feel free to post here.

Cheers
Andreas


From julien.wollbrett at cirad.fr  Tue Nov 25 16:12:49 2008
From: julien.wollbrett at cirad.fr (Julien Wollbrett)
Date: Tue, 25 Nov 2008 17:12:49 +0100
Subject: [MOBY-dev] integration of datatypes with multiparental relations in
 the datatype ontology
Message-ID: <492C2401.7070003@cirad.fr>

I made a plugin to automatically integrate an OWL ontology into the 
datatype ontology.
It's a plugin for prot?g? in which we need to specify both endpoint and
namespace of the specified central registry.
This plugin run with a "simple" ontology (i.e. ontology with no
multiparental relations).
The problem is that I can't create a datatype with multiparental relation.
A lot of biological ontologies have multiparental relations. How can I
integrate these ontologies into the datatype ontology of a specified
central registry?
Did you planned to integrate multiparental relations in biomoby?
Thank you for your help

Julien Wollbrett


From gordonp at ucalgary.ca  Tue Nov 25 18:59:45 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Tue, 25 Nov 2008 11:59:45 -0700
Subject: [MOBY-dev] integration of datatypes with multiparental
 relations in the datatype ontology
In-Reply-To: <492C2401.7070003@cirad.fr>
References: <492C2401.7070003@cirad.fr>
Message-ID: <492C4B21.9050403@ucalgary.ca>

Hi Julien,

The short answer is No. :-)

Can you present a compelling case for multiple inheritance of a 
datatype?  The typical arguments for objects (which have methods, not 
just data) don't apply...

Julien Wollbrett wrote:
> I made a plugin to automatically integrate an OWL ontology into the 
> datatype ontology.
> It's a plugin for prot?g? in which we need to specify both endpoint and
> namespace of the specified central registry.
> This plugin run with a "simple" ontology (i.e. ontology with no
> multiparental relations).
> The problem is that I can't create a datatype with multiparental 
> relation.
> A lot of biological ontologies have multiparental relations. How can I
> integrate these ontologies into the datatype ontology of a specified
> central registry?
> Did you planned to integrate multiparental relations in biomoby?
> Thank you for your help
>
> Julien Wollbrett
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev
>
>


From markw at illuminae.com  Tue Nov 25 19:20:27 2008
From: markw at illuminae.com (Mark Wilkinson)
Date: Tue, 25 Nov 2008 11:20:27 -0800
Subject: [MOBY-dev] [moby] integration of datatypes with multiparental
 relations in the datatype ontology
In-Reply-To: <492C2401.7070003@cirad.fr>
References: <492C2401.7070003@cirad.fr>
Message-ID: <op.uk6v0dhnnbznux@mark.ubic.ubc.ca>

Hi Julien,

I don't see how it would be possible to integrate an "arbitrary" OWL  
ontology into the datatype (you mean Object, right?) ontology in any  
case... Can you please explain in more detail?

At this time, the ISA relationship in the Moby object ontology only allows  
single parenting and there are no plans to change that at the moment.   
This is because the serialization of instances of Moby Objevt nodes is  
tied to the ISA relationship, so it would be a fairly significant change  
in "logic" to allow multiple parenting I suspect...

Mark



On Tue, 25 Nov 2008 08:12:49 -0800, Julien Wollbrett  
<julien.wollbrett at cirad.fr> wrote:

> I made a plugin to automatically integrate an OWL ontology into the  
> datatype ontology.
> It's a plugin for prot?g? in which we need to specify both endpoint and
> namespace of the specified central registry.
> This plugin run with a "simple" ontology (i.e. ontology with no
> multiparental relations).
> The problem is that I can't create a datatype with multiparental  
> relation.
> A lot of biological ontologies have multiparental relations. How can I
> integrate these ontologies into the datatype ontology of a specified
> central registry?
> Did you planned to integrate multiparental relations in biomoby?
> Thank you for your help
>
> Julien Wollbrett
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev




From groscurt at mpiz-koeln.mpg.de  Wed Nov 26 11:19:00 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Wed, 26 Nov 2008 12:19:00 +0100
Subject: [MOBY-dev] New Mobyclient
Message-ID: <492D30A4.1030005@mpiz-koeln.mpg.de>

HI all,

I just checked in a new moby client called FeatureClient. For those of 
you who know my Jabba application 
(http://bioinfo.mpiz-koeln.mpg.de/jabba) - basically its the client i 
use there.

Its a usual moby client with some features:

- Calling services with the concrete objects (like AminoAcidSequence) 
instead of the internal Moby API structures. This helps to understand 
the input and output handling better and to work with full data structures.

- Service discovery based on service name or input/output definition 
(like return all services which consume an AGI locus code and return 
PMIDs). This enables to automatically detect group of services which 
fulfill identical or nearly the same tasks.

- Parallelization for multiple service calls. The client enables you to 
call multiple services at the same time.

- Using a timeout if the service does not return in that time. Limit the 
time you want to wait for a service to return by providing a maximal 
time in which the service shall give back a result.

- Calling a service once with possible different inputs. If you want to 
call one (or more) service(s) with several inputs the client does help you.

if you update your moby you will find more information and how to 
build/use it in the html documentation files and in the README file in 
moby-live/Java/src/support/featureClient

For any concerns,comments or bugs please contact me

Cheers
Andreas



From groscurt at mpiz-koeln.mpg.de  Fri Nov 28 12:08:41 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Fri, 28 Nov 2008 13:08:41 +0100
Subject: [MOBY-dev] Secure (Java) Biomoby web services
Message-ID: <492FDF49.6000702@mpiz-koeln.mpg.de>

Hi,

I have committed changes in the jMoby code to allow people to access 
secure BioMoby web services (this update does not inflict the existing 
code). I also set up a tutorial if someone is interested in creating 
and/or calling a BioMoby secure web service 
(http://www.eu-sol.net/science/bioinformatics/tutorials/secure-biomoby-web-services).

Unfortunately the current tutorial / implementation is only valid for 
Java. It would be great if a Perl person could have a look at it and 
contact me so that we can also set up the system for Perl (the system 
follows basically a usal SSL & HTTP authentication principle).

If you find any errors or have any comments, questions please contact me.

Thanks and cheers
Andreas

-- 
/***************************************************
  Dipl. Bioinf. Andreas Groscurth
  Software developer
  Plant Computational Biology group
  Max-Planck Institute for plant breeding research
  Carl-von-Linne Weg 10
  50829 Cologne
  Germany
  +49(0) 221 5062449
***************************************************/



From dmitry.repchevski at bsc.es  Fri Nov 28 22:40:39 2008
From: dmitry.repchevski at bsc.es (Dmitry Repchevsky)
Date: Fri, 28 Nov 2008 14:40:39 -0800
Subject: [MOBY-dev] Secure (Java) Biomoby web services
In-Reply-To: <492FDF49.6000702@mpiz-koeln.mpg.de>
References: <492FDF49.6000702@mpiz-koeln.mpg.de>
Message-ID: <49307367.7000202@bsc.es>

Hello Andreas,

Last INB meeting (in Malaga) we discussed it too.
The good thing would be to use SSL without a certificate, unfortunately, 
I didn't find a way to configure JBoss in a such way... (?)

The idea is that user don't need to have a certificate to connect to a 
server, but at the same time use SSL encryption to protect its data.

Unfortunately, in your examples I didn't find any configuration of SSL 
on the server side...

In JBoss the configuration is done using
$JBOSS_HOME$\server\$PROFILE$\deploy\jboss-web.deployer\server.xml

*********************************************************
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS"

               ciphers="SSL_DH_anon_WITH_RC4_128_MD5
                        TLS_DH_anon_WITH_AES_128_CBC_SHA
                        TLS_DH_anon_WITH_AES_256_CBC_SHA
                        SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
                        SSL_DH_anon_WITH_DES_CBC_SHA
                        TLS_ECDH_anon_WITH_RC4_128_SHA
                        TLS_ECDH_anon_WITH_AES_128_CBC_SHA
                        TLS_ECDH_anon_WITH_AES_256_CBC_SHA
                        TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA
                        SSL_DH_anon_EXPORT_WITH_RC4_40_MD5
                        SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA
                        TLS_ECDH_anon_WITH_NULL_SHA"

               keystoreFile="${jboss.server.home.dir}/conf/jboss.key"
               keystorePass="PASSWORD_HERE"
               />
*********************************************************

Note the list of supported ciphers (the list is for JDK6, JDK5 has a 
smaller list) - they are anonymous.

The problem is that even SSL allows certificate-free communication 
(using any of those ciphers) JBoss (Tomcat) FAILS if I do not specify a 
keystore...
At the same time on a client side, a client receives a certificate from 
JBoss and fails to find it (obviously).

Another comment is that in your web.xml you don't protect the service 
using SSL, to do so you must specify

***************************************************
<user-data-constraint>
     <transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
***************************************************

I would be happy someone would find a way of using SLL WITHOUT a 
certificates, because you can not expect a user of, say, an applet to 
know how to import them using a keytool.

Kind regards,

Dmitry



From groscurt at mpiz-koeln.mpg.de  Fri Nov 28 14:29:11 2008
From: groscurt at mpiz-koeln.mpg.de (Andreas Groscurth)
Date: Fri, 28 Nov 2008 15:29:11 +0100
Subject: [MOBY-dev] Secure (Java) Biomoby web services
In-Reply-To: <49307367.7000202@bsc.es>
References: <492FDF49.6000702@mpiz-koeln.mpg.de> <49307367.7000202@bsc.es>
Message-ID: <49300037.3040700@mpiz-koeln.mpg.de>

Hi Dimitry,

thanks for your comments. Of course it would be great to have a SSL 
connection without the problem of having and managing the certificates. 
In my tutorial i dont explain the configuration of SSL as this can be 
found on the internet and I dont propose the way of a certificate-less SSL.
But its an interesting idea and I will try to see if I find any solution 
for this and let you know if i find something.... and please keep me 
uptodate if you find something :)


Dmitry Repchevsky wrote:
> Another comment is that in your web.xml you don't protect the service 
> using SSL, to do so you must specify
>
> ***************************************************
> <user-data-constraint>
>     <transport-guarantee>CONFIDENTIAL</transport-guarantee>
> </user-data-constraint>
> *************************************************** 
thanks for pointing that out.

Cheers
Andreas

-- 
/***************************************************
  Dipl. Bioinf. Andreas Groscurth
  Software developer
  Plant Computational Biology group
  Max-Planck Institute for plant breeding research
  Carl-von-Linne Weg 10
  50829 Cologne
  Germany
  +49(0) 221 5062449
***************************************************/



From gordonp at ucalgary.ca  Fri Nov 28 19:38:56 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Fri, 28 Nov 2008 12:38:56 -0700
Subject: [MOBY-dev] jars archive?
Message-ID: <493048D0.8070604@ucalgary.ca>

Hi all,

I think that updates to the jars archive accessible from the Web don't 
work anymore (and perhaps not for some time).  I committed a WAR file 
there last month, and notice it isn't at the Web location for the jars 
archive anymore.  i.e. I was expecting there to be a ...

http://biomoby.org/jmoby-jars/jars-archive/SAWSDLServlet.war

Is the jars-archive update from CVS intentionally defunct?  If so, where 
in CVS should we put all the WARs I create for people to write services?

-Paul


From martin.senger at gmail.com  Sat Nov 29 10:01:50 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Sat, 29 Nov 2008 10:01:50 +0000
Subject: [MOBY-dev] jars archive?
In-Reply-To: <493048D0.8070604@ucalgary.ca>
References: <493048D0.8070604@ucalgary.ca>
Message-ID: <4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>

> I think that updates to the jars archive accessible from the Web don't work
> anymore (and perhaps not for some time).


It was not used since jMoby started to use Maven repository instead.


> Is the jars-archive update from CVS intentionally defunct?


Yes, it is defunct - I have stopped a cronjob that runs the update a while
ago.


>  If so, where in CVS should we put all the WARs I create for people to
> write services?


The question is not much where to put it but how to put it :-) These war
file should end up in the location you mentioned (
http://biomoby.org/jmoby-jars/jars-archive/)<http://biomoby.org/jmoby-jars/jars-archive/SAWSDLServlet.war>-
the location still exists and is accessible from outside.

How do you want to proceed now?

Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger


From gordonp at ucalgary.ca  Sat Nov 29 14:53:09 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Sat, 29 Nov 2008 07:53:09 -0700
Subject: [MOBY-dev] jars archive?
In-Reply-To: <4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>
References: <493048D0.8070604@ucalgary.ca>
	<4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>
Message-ID: <49315755.5050102@ucalgary.ca>

I don't care where I have to put it in CVS, but some directory should be 
copied over to that server regularly.  These WARs are updated not 
infrequently, so a manual copying is not really an option.  Can you 
restore some cron job and let me know where I can put them?

Thanks,

Paul
>>  If so, where in CVS should we put all the WARs I create for people to
>> write services?
>>     
>
>
> The question is not much where to put it but how to put it :-) These war
> file should end up in the location you mentioned (
> http://biomoby.org/jmoby-jars/jars-archive/)<http://biomoby.org/jmoby-jars/jars-archive/SAWSDLServlet.war>-
> the location still exists and is accessible from outside.
>
> How do you want to proceed now?
>
> Martin
>
>   


From martin.senger at gmail.com  Sat Nov 29 17:37:05 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Sat, 29 Nov 2008 17:37:05 +0000
Subject: [MOBY-dev] jars archive?
In-Reply-To: <49315755.5050102@ucalgary.ca>
References: <493048D0.8070604@ucalgary.ca>
	<4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>
	<49315755.5050102@ucalgary.ca>
Message-ID: <4d93f07c0811290937v2a226293w405b37fbffc6e1e3@mail.gmail.com>

> I don't care where I have to put it in CVS


Why should I care? :-)


> Can you restore some cron job and let me know where I can put them?


How often do you want to run the cronjob? What do you want to copy - just
*.war files?

Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger


From gordonp at ucalgary.ca  Sat Nov 29 17:39:59 2008
From: gordonp at ucalgary.ca (Paul Gordon)
Date: Sat, 29 Nov 2008 10:39:59 -0700
Subject: [MOBY-dev] jars archive?
In-Reply-To: <4d93f07c0811290937v2a226293w405b37fbffc6e1e3@mail.gmail.com>
References: <493048D0.8070604@ucalgary.ca>	<4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>	<49315755.5050102@ucalgary.ca>
	<4d93f07c0811290937v2a226293w405b37fbffc6e1e3@mail.gmail.com>
Message-ID: <49317E6F.2060706@ucalgary.ca>

Yeah, just the WAR files, where they are now, once a day would be fine.
>
>
> How often do you want to run the cronjob? What do you want to copy - just
> *.war files?
>
> Martin
>
>   


From martin.senger at gmail.com  Sun Nov 30 10:40:48 2008
From: martin.senger at gmail.com (Martin Senger)
Date: Sun, 30 Nov 2008 10:40:48 +0000
Subject: [MOBY-dev] jars archive?
In-Reply-To: <49317E6F.2060706@ucalgary.ca>
References: <493048D0.8070604@ucalgary.ca>
	<4d93f07c0811290201k1c416a1ch7342310901cab26d@mail.gmail.com>
	<49315755.5050102@ucalgary.ca>
	<4d93f07c0811290937v2a226293w405b37fbffc6e1e3@mail.gmail.com>
	<49317E6F.2060706@ucalgary.ca>
Message-ID: <4d93f07c0811300240m5b4f746bme92feb9da37752e2@mail.gmail.com>

> Yeah, just the WAR files, where they are now, once a day would be fine.


Done. Check here http://biomoby.org/jmoby-jars/jars-archive/ to see what is
available from the jar-archives at the moment.

BTW, for the other developers: If you need some files to be available via
regular HTTP - but you wish to have them also under CVS, you can also use
this 'jar-archive' CVS module. The files put there are passed twice a day to
the URL shown above.

Cheers,
Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger