[Bioperl-l] Skipping/Failing tests

[Mauricio Herrera Cuadra]

Hey Chris,

I don't know if this helps those working on the test suite but, there's 
a recently-cooked recipe for keeping track on the number of tests (thus 
helping to update the test plan accordingly):

http://www.perl.com/pub/a/2007/04/12/lightning-four.html?page=3

My quick .2 cents :)

Cheers,
Mauricio.

Chris Fields wrote:
> To the BioPerl community at large,
> 
> I have noticed a problem with some BioPerl tests when converting to  
> Test::More.  When using the following:
> 
>      while ($seq = $seqin->next_seq) {
>          my $acc = $seq->accession;
>          ok exists $result{ $acc };
>          is $seq->length, $result{ $acc };
>          delete $result{$acc};
>      }
> 
> if $seq is undef then the test plan is off by a factor of 2 for every  
> iteration of the loop.  Two serious problems:
> 
> 1) No specific failures are seen until the end of the test suite when  
> the test plan doesn't match the number of tests (which could be  
> several hundred lines away from the actual failure).
> 2) Worse, if one were lazy enough to not track the actual number of  
> tests (heh, not that would happen) they could inadvertently change  
> the test plan to match the final number of tests.
> 
> There are several ways to work around this, such as using a counter  
> to track the number of iterations and check to make sure they pass:
> 
>      $ct = 0;
>      while ($seq = $seqin->next_seq) {
>          $ct++;
>          my $acc = $seq->accession;
>          ok exists $result{ $acc };
>          is $seq->length, $result{ $acc };
>          delete $result{$acc};
>      }
>      is($ct, 3);
> 
> Here, if $ct is 0 you'll get an error.  However, the test count will  
> still be off at the end (the test plan will be off by 6 tests).
> 
> My opinion is that we should try to match the plan, as a single fail  
> doesn't reflect the severity of the bug (i.e. it should fail each  
> test per iteration, as expected).  Skipping to match is an option as  
> well (one I've used) but again doesn't reflect the severity of the  
> problem in my opinion.  The flip side is that some consider any  
> failed test significant, so there is no reason to try matching the  
> tests up.
> 
> What I would like to do is hammer out something we can add to the  
> Writing Tests HOWTO which addresses some ways to deal with the above  
> for those who want to contribute code and tests to BioPerl.  I'm  
> looking for some (any) additional opinions on the matter (or, if you  
> have the initiative, adding some ideas to the HOWTO itself).
> 
> http://www.bioperl.org/wiki/Special:Recentchanges
> 
> Thanks!
> 
> chris
> 
> _______________________________________________
> Bioperl-l mailing list
> Bioperl-l at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/bioperl-l
> 

-- 
MAURICIO HERRERA CUADRA
arareko at campus.iztacala.unam.mx
Laboratorio de Genética
Unidad de Morfofisiología y Función
Facultad de Estudios Superiores Iztacala, UNAM