[Bioperl-l] Tests involving remote databases

[Chris Fields]

I'm not sure how Bio::Root::HTTPget deals with errors, but you can  
test for several conditions when using LWP (success, error, etc) and  
throw an error accordingly.  So you could possibly test $@ with a  
regex for the error message, pass/fail tests based on that, followed  
by skipping further tests.  Test::More has like/unlike to deal with  
that; you could use plain regex using Test.pm.

SKIP:{
   skip('Set BIOPERLDEBUG=1 to run tests which require remote DB  
access', 5)
       if !$DEBUG;
   my $db = Bio::DB::GenBank->new();
   my $seq;
   eval { $seq = $db->get_Seq_by_acc('ABC123')};
   if ($@) {
       # test to make sure the URL is unreachable
       like{$@, qr{bad hostname}, 'URL test'};
       # must skip tests regardless, anyway
       skip('Bio::DB::GenBank URL problem', 4);
   } else {
       ok(!$@, 'Bio::DB::GenBank URL test');
   }
   ... # four more tests based on $seq
}

Caveat: haven't tried the above in the real world, but it should work.

Having a set of instructions for tests is a good idea, maybe  
something for the wiki and developers, similar to Torsten's Best  
Practices page.  Working code from tests would be a good start.

chris

On Sep 29, 2006, at 3:40 PM, Hilmar Lapp wrote:
> Can we include a library function (or instructions) somewhere to  
> use for test writers that will determine based on the error string  
> whether merely the network is out or whether the URL is unreachable  
> (supposedly because it's wrong).
>
> 	-hilmar
>
> On Sep 29, 2006, at 12:44 PM, Chris Fields wrote:
>
>> Sendu, all,
>>
>> We're running into several problems with tests being skipped based  
>> on URL
>> failure but passing as 'ok'.  Two incidents involving XEMBL_DB.t and
>> Biblio_biofetch.t come to mind, and recent problems have now  
>> surfaced with
>> proxies (thanks Torsten!).
>>
>> As one option, I would like to propose using the following SKIP  
>> block format
>> (or something similar) and Test::More to run checks on remote DB's  
>> in the
>> various tests where remote DB access is required.  There are probably
>> similar ways to accomplish the same thing using Test, but I believe
>> Test::More makes it easier.
>>
>> SKIP:{
>>   skip('Set BIOPERLDEBUG=1 to run tests which require remote DB  
>> access', 5)
>>       if !$DEBUG;
>>   my $db = Bio::DB::GenBank->new();
>>   my $seq;
>>   eval { $seq = $db->get_Seq_by_acc('ABC123')};
>>   ok(!$@, 'Bio::DB::GenBank URL test');
>>   skip('Bio::DB::GenBank URL failure', 4) if $@;
>>   ... # four more tests based on $seq
>> }
>>
>> Most of us do not run each set of tests individually unless we are
>> developing code that relies on a specific set.  Most often, when  
>> we run all
>> of the tests we use 'make test' and Test::Harness, which treats  
>> skipped
>> tests as 'ok.'
>>
>> So, in effect, we never see any URL failure, just that all tests  
>> pass.  This
>> practice isn't used by other CPAN modules.  WWW::Shorten, for  
>> instance, runs
>> tests on all URLs and fails if they are invalid.  If we did the  
>> same thing,
>> we would have picked up on the following almost immediately when  
>> running
>> full tests:
>>
>> 1)  The XEMBL server has been out for over six months.
>> 2)  The Biblio_biofetch.t tests probably never worked correctly,  
>> judging by
>> recent fixes (thanks Brian).
>>
>> However, we would never had known that if we relied strictly on  
>> the summary
>> results from running 'make test.'  Blindly skipping these doesn't  
>> inform us
>> when the URL is invalid, and so we never manage to address the  
>> issue when it
>> pops up.  Explicit and consistent test failures let us know when  
>> things go
>> wrong (i.e. when the URL is no longer valid).
>>
>> Furthermore, the debugging output that accompanies the tests when  
>> using
>> BIOPERLDEBUG=1 obfuscates any potential error messages, so we tend  
>> to miss
>> warning flags that pop up (such as the biblio_biofetch.t warning  
>> message
>> about the bad URL).  Torsten has a proposal that we use a  
>> different variable
>> for running remote DB tests, which IMHO we should consider and  
>> which should
>> take care of this.
>>
>> Setting these to run based on BIOPERLDEBUG=1 also passes over the  
>> remote
>> tests for most users, which shouldn't cause a problem with  
>> spamming the
>> servers.
>>
>> Thoughts?  Flames?
>>
>> Chris
>>
>> Christopher Fields
>> Postdoctoral Researcher - Switzer Lab
>> Dept. of Biochemistry
>> University of Illinois Urbana-Champaign
>>
>>
>> _______________________________________________
>> Bioperl-l mailing list
>> Bioperl-l at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>
>
> -- 
> ===========================================================
> : Hilmar Lapp  -:-  Durham, NC  -:-  hlapp at gmx dot net :
> ===========================================================
>
>
>
>
>

Christopher Fields
Postdoctoral Researcher
Lab of Dr. Robert Switzer
Dept of Biochemistry
University of Illinois Urbana-Champaign