[BioRuby] rq: Zero configuration job scheduler for computer clusters and multi-core

[Pjotr Prins]

Use those cores!

I just created a functional gem for rq, a job scheduler created by
Ara Howard. It allows running jobs in parallel, without any
configuration (just a shared directory between processes/machines).

QUICK START

  install rq using rubygems, after installing sqlite 2.x (dev version, 
  on Debian apt-get install libsqlite0-dev)

    gem1.8 install rq-ruby1.8

  the binary is in /var/lib/gems/1.8/bin/, so add that to the path,
  or create a symbolic link

    ln -sf `gem1.8 contents rq-ruby1.8|grep bin/rq$` /usr/local/bin/rq

  now rq should work

    rq --help

  run the integration test

    /var/lib/gems/1.8/bin/test_rq.rb

  set up a directory for your queue - this can be a local, or an NFS/sshfs
  mounted drive:

    rq dir create

  on every node create a queue runner, specifying the number of cores (here 8)

    rq dir feed --daemon --log=rq.log --max_feed=8

  submit two jobs - shell style
   
    rq dir submit 'sleep 10'
    rq dir submit 'sleep 9'

  check status

    rq dir status

  shows

        --- 
        jobs: 
          pending: 0
          holding: 0
          running: 2
          finished: 0
          dead: 0
          total: 2
        temporal: 
          running: 
            min: {2: 00h00m03.49s}
            max: {1: 00h00m03.60s}
        performance: 
          avg_time_per_job: 00h00m00.00s
          n_jobs_in_last_hrs: 
            1: 0
            12: 0
            24: 0
        exit_status: 
          successes: 0
          failures: 0
          ok: 0

  Now, that was easy!!

rq will be a standard feature of the BioLinux VMs.

I also ported the code to Ruby1.9 - it is sitting in the ruby1.9 branch on github:

  https://github.com/pjotrp/rq/tree/ruby1.9

it would be good if others were to test that in some production setup,
before I push that to rubygems.

Pj.