[Bioperl-pipeline] multiple pipelines

[jeremyp at sgx3.bmb.uga.edu]

Hi,

>
> ...finally the web manager would track multiple pipelines. This is at
> discussion stage at the moment, though Juguang and Aaron over here
> seem set to work on this soon.

Yes, these ideas seem great. I would personally put in a vote for at
least having  pure cgi as an option (as opposed to only having a Java
based client for example).

But, for now, how safe is it to run two pipelines at once? Especially,
has anyone done any workarounds to allow the PipelineManager to write to
different tmp directories? (if not, I I will something very simple to
keep the execution scripts separate)

>
> Absolutely. To achieve best performance you need:
>
> 1-Blast database local to node with best possible read speed (in our
> case with 2  mirrored local hard disks)

I don't know if you have any numbers or not, but I wonder what the
approximate percent speed gain is from doing this... any idea? That is
obviously a very aggressive setup... the type of setup I would expect on
a heavily used/publicly accessible resource.

>
> 2-Write STDOUT and STDERR to local node, read results from there and
> finally store results in database (no need to copy anywhere)
>
> The only current caveat with point 2 is that if a job fails, the error
> file stays there...


So, is doing this included in the current code? I didn't notice this...
or is it not there due to the problem you mentioned.

Actually, initially, I was doing basically this. I set NFSTMP_DIR to
/tmp, which is local on each machine. But, I had to stop doing that when
the pipeline started making subdirectories in NFSTMP_DIR. I think the
pbs software was automatically copying (scp) the output to /tmp on the
master node... I'm not exactly sure how that was working though.

Jeremy