[Bioperl-l] dealing with large files

[Chris Fields]

On Dec 20, 2007, at 11:06 AM, Sendu Bala wrote:

> Chris Fields wrote:
>> The only way I can think of to fix this would be (as Jason also  
>> suggested) lightweight objects, or something like the lazy sequence  
>> object ala the SwissKnife suite (which only bring what you want  
>> into memory).
>> Related to that, I have been testing something like that, which  
>> uses iterators to pass in chunks of data from a stream to handlers  
>> to build a sequence object.  Wouldn't be too hard to reconfigure  
>> that to return file positions as well.  Maybe for the 1.7 release...
>
> Bio::PullParserI is your friend.

I'm looking into that, yes.  I'm thinking of something like a generic  
lazy sequence class with an embedded Handler/PullParser object which  
processes stuff on the fly.

Oh, when I have a bit more time...

chris