[Bio-packaging] Using a shared Guix store (was RE: testing out guix)

[Cook, Malcolm]

Ricardo, Pj, et al,


... Hi - I am resending this message as I omitted Guix-devel <guix-devel at gnu.org> at first and would like to cast a broader net....


I am interested in understanding details behind Ricardo's observation: "Guix is not designed to be run in a centralised manner. A Guix daemon is supposed to run on each system as root and it listens to RPCs from local users only. In an environment with multiple clusters and multiple workstations this approach requires considerable effort to make it work correctly and securely. Instead we opted to run the Guix daemon on a single dedicated server, writing profile data and store items onto an NFS share. . The cluster nodes and workstations mount this share read-only." (http://elephly.net/posts/2015-04-17-gnu-guix.html)

Can anyone elaborate a little on what are the obstacles to having  `/gnu` mounted read-write network wide?

Can it be partially characterized as:

	Multi-process contention over un-coordinated access to the store (especially write access necessitated by supporting the `build` action)

If so, might this be mitigated using a variant off of "Using the Offload Facility" (http://www.gnu.org/software/guix/manual/guix.html#Daemon-Offload-Setup) in which builds would still be offloaded (and thus subject to coordination), with the elimination of the need for " Missing prerequisites for the build are copied over SSH to the target machine, which then proceeds with the build; upon success the output(s) of the build are copied back to the initial machine" since they would be done through the shared file system?

Do I understand correctly that in your setup, Ricardo, that absolutely no `guix` commands are executed on any host other than the "single dedicated server".   What about `guix environment p1 p2 p3` when p1 p2 p3 are already available in /gnu.  If I understand correctly, in such a case, nothing need be written to /gnu... and so should not present any challenge to running guix off a shared mount.   Or am I missing an aspect of what is going on?

Lacking the ability to, from any host, dynamically alter the runtime environment to include _already installed_  scientific applications seems like the most important aspect of guix that would be untolerable to my users were I to user guix.    I do hope I am mis-understanding the trade-off you made at your site.

I think the second-worst trade-off this compromise takes is that a user cannot even alter their own profile unless connected to the master guix host.  For instance, a user switching her default emacs  between two already built & installed versions only requires editing the profiles/per-user symlink farm; couldn't such commands be put behind a per-user semaphore,  allowing guix/profiles/per-user to be made available +rw on a shared network mount?

I really appreciate everyone's assistance in helping me understand these trade-offs, how guix works, and if there is any new or different thinking about strategies for deploying guix.

Thanks,

Malcolm Cook



Hi Malcolm,

> Pleased to e-meet you.  It was your
> http://elephly.net/posts/2015-04-17-gnu-guix.html that got on this 
> path.  I'm so glad I found it.

Oh, great to read that it reached someone who might benefit from it :)

> Great.  I have a trove of recipes (in either bash and a
> brew-derivative) which I intend to move into guix over time, unless 
> I'm beaten to any of them (I'm sure I already have been for some):>

[...]

I just went through the list and found a few that looked familiar.
Below are some comments.

Non-free, won't package:

> gatk
> jimKentUtils

On my list (possibly non-free):

> igv
> igvtools
> meme
> picard (requires more Java toolchain stuff to build from source)
> soapdenovo2
> trimmomatic

Non-free stuff, packaged in my private guix-nonfree repo:

> macs14
> tophat
> viennaRNA

Free stuff, packaged and available through Guix:

> bamtools
> bedtools
> blast+  (WIP, it's really big)
> bowtie2
> cufflinks
> fastqc
> fastx_toolkit
> graphviz
> ncbi_sra_toolkit
> ngsutils
> rsem
> star
> tabix

These look familiar but I'm not sure if I already have them:

> bam2fastq
> bcftools

[...]

> Gonna be a party!

Indeed!  I would be very happy if you decide for Guix in the end.  There isn't a lot of bioinfo stuff there yet, but that can be fixed by having more people contribute recipes.

Cheers,
Ricardo