[Bioperl-pipeline] Re: BioPipe

[Juguang Xiao]

>
>> Currently, one manager is in charge of one running pipe. if you want 
>> to run more, just start up more managers, why do not need one manager 
>> handlers many pipes?
>
> Because it is extremely counter-intuitive to have to run multiple 
> applications to run many pipelines. Just like the whole cluster is 
> managed with one application, so should pipelines. This is not only a 
> user interface issue. This is also a change to the schema, to handle 
> multiple pipelines in one database, rather than many databases, and 
> have pipeline ids on the jobs, on the files,etc. etc.
>

One thing I need to remind is about Perl's thread. As I gave up the 
Bio::DB::BioSQL::MultiDB using thread, thread-version perl raise more 
problems than its benefit. If you want to implement the above 
mentioned, you bet the Perl thread.

If the thread problem cannot be solved in perl, you cannot make 
pipeline manager server program access multiple connection as well.

My suggest is to let pipeline managers running as process and each has 
one database. On the top of it, there builds a server program to 
connect with client and simply submit the process on the cluster. Due 
to the perl thread problem, if it is true, we should use Java. We do 
not need a lot intercommunication between the Java marshall and the 
perl's pipeline managers, I think.

my $.02

Juguang