[Bioperl-pipeline] Re: ciona pipeline

[Elia Stupka]

Hello guys,

good to hear the pipeline was "humming away" while we had our weekends,
nothing better than seeing the computers sweat while you rest from time to
time :)

> So far we only have about 17 failed jobs. All  result from connection to
db problems

That is already great news. We could up the number of allowed connections,
but in any case 17 failures is great news!

> 1) Need to batch the number of jobs that are created/stored so we don't
have to wait
> for all before submitting them. this is trivial I think. The create
initial jobs
> part takes too long. .

Sure, that's the part I was chatting about the other day, we should
definitely start submitting after a few hundred are created, and we should
do that as a function of the number of nodes available.

> sure this doesn't happen but we pickoff from where it stopped and
> be able to run the Pipeline in either a fresh/continue mode.

Oooppss had never noticed this, this is quite serious and should be fixed...

> 3) Similar to Ensembl, have a pipeline lock file that just ensures that
while a pipeline

Very sane ;)

Elia