[Bioperl-l] perl memory problem

[Barry Moore]

Zaigang-

Tie::File will allow you to tie a file to an array.  You can then step
through the array, modifying it as you go if you like, and only the
portion of the file that you are working on is actually current in
memory.  It's easy to use and has worked well for me in similar
situations.

Barry

> -----Original Message-----
> From: bioperl-l-bounces at portal.open-bio.org [mailto:bioperl-l-
> bounces at portal.open-bio.org] On Behalf Of Jason Stajich
> Sent: Wednesday, October 26, 2005 11:30 AM
> To: Zaigang Liu
> Cc: bioperl-l at portal.open-bio.org
> Subject: Re: [Bioperl-l] perl memory problem
> 
> 
> On Oct 26, 2005, at 11:55 AM, Zaigang Liu wrote:
> 
> >
> >
> >
> >
> > Hello, everyone,
> >
> > I am new to this list and I am looking for help for one problem I
met.
> >
> > I am trying to run a perl program doing calculation on large data
> > set. It
> > uses an array to store the input data and the array size will reach
8G
> > after all the data is loaded by my estimation. The process always
> > stops and
> > gives a "out of memory" error when the memory usage reached about
> > 2G. I
> > used a solaris machine with 40G memory so it is not really run out
of
> > memory when the program stopped.
> >
> there is probably a limit on how much memory perl can address in the
> version you have compiled.
> 
> Have you considered using a data structure which does not require the
> entire array to be in memory at once.  the DB_File module is very
> useful for this as you can tie your array to a flatfile Berkeley DB.
> You can also use something like SQLite or a full fledged relational
> database instead if you are crunching such a large number of items.
> 
> 
> > Does anyone know where could this program come from and how to fix
it.
> >
> > Thanks in advance,
> >
> > Zaigang
> >
> > This communication is for use by the intended recipient and contains
> > information that may be Privileged, confidential or copyrighted
under
> > applicable law. If you are not the intended recipient, you are
hereby
> > formally notified that any use, copying or distribution of this e-
> > mail,
> > in whole or in part, is strictly prohibited. Please notify the
> > sender by
> > return e-mail and delete this e-mail from your system. Unless
> > explicitly
> > and conspicuously designated as "E-Contract Intended", this e-mail
> > does
> > not constitute a contract offer, a contract amendment, or an
> > acceptance
> > of a contract offer. This e-mail does not constitute a consent to
the
> > use of sender's contact information for direct marketing purposes
> > or for
> > transfers of data to third parties.
> >
> > Francais Deutsch Italiano  Espanol  Portugues  Japanese  Chinese
> > Korean
> >
> >            http://www.DuPont.com/corp/email_disclaimer.html
> > _______________________________________________
> > Bioperl-l mailing list
> > Bioperl-l at portal.open-bio.org
> > http://portal.open-bio.org/mailman/listinfo/bioperl-l
> >
> 
> --
> Jason Stajich
> Duke University
> http://www.duke.edu/~jes12
> 
> 
> _______________________________________________
> Bioperl-l mailing list
> Bioperl-l at portal.open-bio.org
> http://portal.open-bio.org/mailman/listinfo/bioperl-l