[Bioperl-pipeline] ciona pipeline

[shawnh@worf.fugu-sg.org]

Hi guys,
	the ciona pipeline has been running since yesterday. RepeatMasker and Blast jobs
are being created and running. Currently we have about 20,000 jobs in the queue.
So far we only have about 17 failed jobs. All  result from connection to db problems
which result from too many connections. Something I don't think should have been a 
problem. Might it be cases where we recevie heavy web activiy? Nevertheless,
its not a problem since jobs will just be re-submitted.

So small changes that I'm working on or kiran could think about is the PipelineManager.
1)	Need to batch the number of jobs that are created/stored so we don't have to wait
for all before submitting them. this is trivial I think. The create initial jobs
part takes too long. . 
2)	THe other thing is that create_initial_jobs which I suspect will change
soon, is that the way _create_initial_jobs is done, it doesn't support stoppage of
the pipeline and re-running. It will re-create the inputs/jobs everytime the pipeline 
is run...we need to make
sure this doesn't happen but we pickoff from where it stopped and
be able to run the Pipeline in either a fresh/continue mode.
3) 	Similar to Ensembl, have a pipeline lock file that just ensures that while a pipeline
a particular pipeline is running, no other pipelinemanager will run on that same pipeline so we
don't conflict and overwrite stuff. For testing purposes, we just have an option that flushes all
lock files. Since we don't have this yest...please ensure no one writes/test on Ciona_Pipe2 and EnsemblTest3





shawn

 

-- 
'Torture the data long enough and they will reveal anything'
********************************
* Shawn Hoon
* http://www.fugu-sg.org/~shawnh
********************************