[Bioperl-l] Skipping/Failing tests

[Chris Fields]

To the BioPerl community at large,

I have noticed a problem with some BioPerl tests when converting to  
Test::More.  When using the following:

     while ($seq = $seqin->next_seq) {
         my $acc = $seq->accession;
         ok exists $result{ $acc };
         is $seq->length, $result{ $acc };
         delete $result{$acc};
     }

if $seq is undef then the test plan is off by a factor of 2 for every  
iteration of the loop.  Two serious problems:

1) No specific failures are seen until the end of the test suite when  
the test plan doesn't match the number of tests (which could be  
several hundred lines away from the actual failure).
2) Worse, if one were lazy enough to not track the actual number of  
tests (heh, not that would happen) they could inadvertently change  
the test plan to match the final number of tests.

There are several ways to work around this, such as using a counter  
to track the number of iterations and check to make sure they pass:

     $ct = 0;
     while ($seq = $seqin->next_seq) {
         $ct++;
         my $acc = $seq->accession;
         ok exists $result{ $acc };
         is $seq->length, $result{ $acc };
         delete $result{$acc};
     }
     is($ct, 3);

Here, if $ct is 0 you'll get an error.  However, the test count will  
still be off at the end (the test plan will be off by 6 tests).

My opinion is that we should try to match the plan, as a single fail  
doesn't reflect the severity of the bug (i.e. it should fail each  
test per iteration, as expected).  Skipping to match is an option as  
well (one I've used) but again doesn't reflect the severity of the  
problem in my opinion.  The flip side is that some consider any  
failed test significant, so there is no reason to try matching the  
tests up.

What I would like to do is hammer out something we can add to the  
Writing Tests HOWTO which addresses some ways to deal with the above  
for those who want to contribute code and tests to BioPerl.  I'm  
looking for some (any) additional opinions on the matter (or, if you  
have the initiative, adding some ideas to the HOWTO itself).

http://www.bioperl.org/wiki/Special:Recentchanges

Thanks!

chris