[Bioperl-l] Re: [Bioclusters] BioPerl and memory handling

[Michael Maibaum]

On Tue, Nov 30, 2004 at 01:24:24AM -0800, Steve Chervitz wrote:
>Regarding SearchIO memory usage, I don't think this has been an issue
>before, so I wonder if there is something about the installation or specific
>usage of it that is leading to memory hogging. I've run it over large
>numbers of reports without noticing troubles. It would be useful to see a
>sample report + script using SearchIO that leads to the memory troubles, so
>we can try to reproduce it.


FWIW - I at least didn't have a problem parsing many thousands of results in a stram with SearchIO - I had a problem with parsing certain specific result sets, Essentially anything with about 2000 hits and alignments (or more) for a single query would kill a linux box with 1 gig of RAM (it would thrash VM to death). These would run on a opteron 16Gig box and used >8 gig of RAM in some cases. 

As far as I can see the majority of the memory was then returned when BioPerl moved on to the next record. The issue is that it takes a rather large amount or RAM for an individual record and I assumed (rightly or wrongly) that BioPerl slurps up the entire record and builds the objects representing it as a whole hence the large RAM usage. It may be that the objects to represetn 2000+ hits are just very (unreasonably?) large.

Michael