[Open-bio-l] Automated testing server; was [BioRuby] tutorial

[Peter Cock]

On Fri, Mar 4, 2011 at 2:14 PM, Chris Fields <cjfields at illinois.edu> wrote:
> On Mar 4, 2011, at 5:45 AM, Peter Cock wrote:
>> On Thu, Mar 3, 2011 at 8:52 PM, Chris Fields <cjfields at illinois.edu> wrote:
>>> On Mar 3, 2011, at 2:29 PM, Raoul Bonnal wrote:
>>>
>>>> Dear All,
>>>> a) this discussion is a good starting point for a workgroup during Codefest
>>>> 2011 and/or BOSC (can't post to bopen-bio list cause I don't have the addr
>>>> here, sorry) a.idea) why not create a common "machine" on the cloud for
>>>> testing our projects ?! It would be fun to see all the bio* with stats about
>>>> testing etc...
>>>
>>> I think the buildbot instance covers some of that, but they are mainly
>>> automated builds:
>>>
>>> http://testing.open-bio.org/
>>>
>>> biopython is the only one set up currently.
>>
>> Note that this is a small Amazon cloud machine, and doesn't actually run
>> the tests itself (that would require a more expensive machine due to the
>> higher load). Instead we need client machines (covering a range of OS,
>> Python version etc) whose run-time is donated (currently by Biopython
>> developers or their employers).
>>
>> When setting up http://testing.open-bio.org/ the intention was to have
>> it host other Bio* projects too - and they don't have to be using the
>> buildbot software (but they could if they wanted to).
>
> I think it would be nice to have a smoke server there as well (I don't
> think buildbot can do that, AFAIK).  Will ask Chris D about it.

Can you clarify what you mean by smoke server? I'd assumed it was
a product name but now it sounds like you mean some methodology.

We have buildbot setup to do a nightly test of the latest code on all
the slaves. Additionally, via the web interface an admin can force a
rebuild, and force a build of the current latest code.

In theory one can also hook up buildbot to run tests triggered by
commits - I understand this isn't available for git repositories yet
though (at least, that's my recollection from dicussion with Tiago,
BCC'd in case he isn't on open-bio-l yet).

> I like the fact that we now are diversifying beyond perl-based
> solutions with buildbot and redmine (I would include mediawiki,
> but PHP gives me the hives :).  Any easily-installable Java-based
> ones?

As long as the OBF has enough people familiar with the tool
to support it, I don't care what language it is written in.

>>>> b) testing is very important and new contributes should be accepted
>>>> only with tests, which implies a problem how to check fake tests? :-)
>>>
>>> Code review and checking the test results.  Otherwise, you have to trust
>>> your developers.
>>
>> This is something we're pushing much more with Biopython in recent
>> years (a depressing amount of legacy code had no tests when I first
>> got involved, things are a lot better now). For any new code contributions
>> we do ask for tests and minimal documentation (and try and work
>> with the contributor to help this happen).
>
> Same here.  This is a common cycle with OS projects.
>
>>>> c) usually ruby's community relies a lot on doc==tests that's
>>>> completely wrong, because the newbies can't read fluently the
>>>> code, also because there are different tools and sometimes a lot
>>>> of mocking.
>>
>> With Biopython we're starting to make more use of testable docs
>> (Python's doctest system) which is really good I think. But you are
>> right, you need proper unit tests AS WELL. With our doctests,
>> the primary aim is to be useful documentation (which can double
>> as a basic functional test).
>
> The problem with bioperl is that much of the testing framework
> for in-line or testable docs wasn't present during the initial
> development process.  We also have a LOT of cruft modules
> that lack true author support.

Also true for Biopython to some extent.

> What we are planning on for future releases is to modularize
> bioperl installations, with an installer to put together the dists
> one wants.  The current release has well over 1000 classes
> and modules; modularizing will help remove some of the
> maintenance headaches we had been facing during
> development.  We are also pushing new code to go into
> separate CPAN installations, as these can be added to an
> installer as needed (as someone else long ago put it, we're
> essentially creating a mini-BioPerlPAN within CPAN).

I'm happy enough with the current Biopython monolith install -
although we are slowly gaining extra runtime dependencies,
doing the install/build is pretty easy. As the Python packaging
infrastructure matures, we too may want to consider
modularising - but for now I feel that is premature.

There is also an issue here for testing - if you can have a
diverse set of build slaves, then hopefully some will have
and some will lack each soft dependency. This is useful to
check graceful failure when things are missing.

Peter