[Biopython] multiprocessing problem with pysam

Michal mictadlo at gmail.com
Sun May 15 05:35:07 UTC 2011


Hello,
Thank you Brad. I have written the following new code:

import re
import os
import pysam
from pprint import pprint
from multiprocessing import Pool


class Test():

     def __init__(self, bam_filename, cultivars):
         self.__bam_fh = pysam.Samfile(bam_filename, "rb")
         self.__cultivars = cultivars

     def run(self, ref_name):
         print os.getpid(), ref_name, self.__cultivars
         return (os.getpid(), ref_name)


if __name__ == '__main__':
     cultivars = 'Ja,Ea,As'.replace(' ', '').split(',')
     bam_filename = "/media/usb/tests/test.bam"

     bamfile = pysam.Samfile(bam_filename, "rb")

     ref_names = bamfile.references
     ref_lengths = bamfile.lengths
     bamfile.close()

#    for ref_name in ref_names:
#        Test(bam_filename, cultivars).run(ref_names)


     pool = Pool()
     results = dict(pool.imap_unordered(
         Test(bam_filename, cultivars).run, ref_names))
     pool.close()
     pool.join()
     pprint(results)


and got the follwing error:

Exception in thread Thread-2:
Traceback (most recent call last):
   File "/home/mictadlo/apps/python/lib/python2.7/threading.py", line 
530, in __bootstrap_inner
     self.run()
   File "/home/mictadlo/apps/python/lib/python2.7/threading.py", line 
483, in run
     self.__target(*self.__args, **self.__kwargs)
   File 
"/home/mictadlo/apps/python/lib/python2.7/multiprocessing/pool.py", line 
285, in _handle_tasks
     put(task)
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup 
__builtin__.instancemethod failed


I have search and found two possible solution for this problem:
* http://www.doughellmann.com/PyMOTW/multiprocessing/communication.html
* http://www.rueckstiess.net/research/snippets/show/ca1d7d90

However, is there a better way to solve it or the above solution are not 
good?

Thank you in advance.

Michal


















More information about the Biopython mailing list