[Open-bio-l] Automated testing server; was [BioRuby] tutorial

[Chris Fields]

On Mar 4, 2011, at 9:04 AM, Peter Cock wrote:

> On Fri, Mar 4, 2011 at 2:14 PM, Chris Fields <cjfields at illinois.edu> wrote:
>> On Mar 4, 2011, at 5:45 AM, Peter Cock wrote:
>>> On Thu, Mar 3, 2011 at 8:52 PM, Chris Fields <cjfields at illinois.edu> wrote:
>>>> On Mar 3, 2011, at 2:29 PM, Raoul Bonnal wrote:
>>>> 
>>>>> Dear All,
>>>>> a) this discussion is a good starting point for a workgroup during Codefest
>>>>> 2011 and/or BOSC (can't post to bopen-bio list cause I don't have the addr
>>>>> here, sorry) a.idea) why not create a common "machine" on the cloud for
>>>>> testing our projects ?! It would be fun to see all the bio* with stats about
>>>>> testing etc...
>>>> 
>>>> I think the buildbot instance covers some of that, but they are mainly
>>>> automated builds:
>>>> 
>>>> http://testing.open-bio.org/
>>>> 
>>>> biopython is the only one set up currently.
>>> 
>>> Note that this is a small Amazon cloud machine, and doesn't actually run
>>> the tests itself (that would require a more expensive machine due to the
>>> higher load). Instead we need client machines (covering a range of OS,
>>> Python version etc) whose run-time is donated (currently by Biopython
>>> developers or their employers).
>>> 
>>> When setting up http://testing.open-bio.org/ the intention was to have
>>> it host other Bio* projects too - and they don't have to be using the
>>> buildbot software (but they could if they wanted to).
>> 
>> I think it would be nice to have a smoke server there as well (I don't
>> think buildbot can do that, AFAIK).  Will ask Chris D about it.
> 
> Can you clarify what you mean by smoke server? I'd assumed it was
> a product name but now it sounds like you mean some methodology.
> 
> We have buildbot setup to do a nightly test of the latest code on all
> the slaves. Additionally, via the web interface an admin can force a
> rebuild, and force a build of the current latest code.
> 
> In theory one can also hook up buildbot to run tests triggered by
> commits - I understand this isn't available for git repositories yet
> though (at least, that's my recollection from dicussion with Tiago,
> BCC'd in case he isn't on open-bio-l yet).

A smoke server holds test results submitted from users, or really anyone who has a set of modules/tools installed; it can be part of the build/testing process.  It's less automated building and more distributed than buildbot: the server doesn't call out to a specific set of build servers to run builds and tests, but relies on the community (developers, users) to run tests and submit them.  CPAN has this built in via CPAN Testers; one can install a set of modules and then, every time a dist is installed via CPAN a test report is sent to a remote server:

http://www.cpantesters.org/distro/B/bioperl.html

This is similar to what the bioruby folks are searching for, if I'm not mistaken.  

(Just so there is no confusion, in most cases with software 'smoke testing' refers to a set of prelim tests to make sure a build works, but in practice with small projects the term equally applies to running every test)

>> I like the fact that we now are diversifying beyond perl-based
>> solutions with buildbot and redmine (I would include mediawiki,
>> but PHP gives me the hives :).  Any easily-installable Java-based
>> ones?
> 
> As long as the OBF has enough people familiar with the tool
> to support it, I don't care what language it is written in.

Same here.

chris