[Bioperl-pipeline] [Question] Pipeline vs. PBS - the SAME question posted to Pipeline Mailing List

[Lenny Teytelman]

Dear Sang Chul Choi,

Let me answer the second question first, and that should clear things up.

All the machines are connected through NFS.  So, even though
programs, bioperl, etc. might only be installed on the server, they are
installed in NFS-visible directories and can be seen from each node.

Indeed, BioPipe, BioPerl, MySQL, fasta databases, and executables are all
on the server, but they are all visible from each node.

The same goes for the environment variables.  I only had to define them
once in the .cshrc in my home directory, for them to be in force all the
machines.  In any case, you definitely need to have them set on all the
nodes.

As for NFS, you can see that everything in our setup is based on it.  If I
am not mistaken, the Singapore guys are also using NFS for their
computational jobs.  At the same time, the Ensembl pipeline people loathe
NFS and swear that it's not stable enough.  This depends on how big your
compute databases are and how many machines you are using.  At some point,
you may need to push all the big files across all of the nodes rather than
have them available through NFS.

Hope this is helpful,

Lenny

> 1. Environment Variables > ---------------------------
> In BioPipeline Prerequisites, you said that you had added the following 
> environment variables into some machines.
> > 5) Environment variables
> > #In .cshrc add:
> > setenv PIPEHOME /u/blue2/BioPipe
> > setenv PERL5LIB
> > /u/blue2/BioPipe/bioperl-live:/u/blue2/kiran/src/bioperl-pipeline
> > setenv PATH /usr/pbs/bin/:${PATH}
> > ------------------
> 
> My question is: which machines did you add those variables into?
>       a. Server only
>       b. Node only
>       c. Server & Nodes
> 
> 2. Could you explain your system architecture layout of the setup?
> I think that there are serveral machines like this:
> <Server>
> bioperl-live, bioperl-run, bioperl-pipeline, program(BLAST, repeatmasker, etc),
> DBD, DBI, pbs_server, pbs_sched, pbs_mom
> 
> <Nodes>
> DBD, DBI, pbs_mom
> 
> <DB Server>
> MySQL
> 
> Sincerely yours, 
> 
> Sang Chul Choi
> 
> ============================= 
> Live, Love, and Learn!