[Bioperl-l] load_seqdatabase.pl memory requirements unusually large

Mon Feb 9 20:36:24 UTC 2009

On Feb 6, 2009, at 1:34 PM, Hilmar Lapp wrote:

> Something seems to cause Perl to be crapping out. It it were a  
> programmatic exception you would see the message and the trace.
>
> Could you run these tests by themselves:
>
> 	./Build test --test-files t/11locuslink.t
>
> If that doesn't reveal the error, add a verbose=1 argument.
>
> Let us know what you find.

Here's the result:

[root at srv-cf1 bioperl-db]# ../Build test --test-files t/11locuslink.t
bash: ../Build: No such file or directory
[root at srv-cf1 bioperl-db]# ./Build test --test-files t/11locuslink.t
Copying scripts/biosql/terms/importrelation.pl -> blib/script/ 
importrelation.pl
blib/script/importrelation.pl -> blib/script/bp_importrelation.pl
Copying scripts/biosql/merge-unique-ann.pl -> blib/script/merge-unique- 
ann.pl
blib/script/merge-unique-ann.pl -> blib/script/bp_merge-unique-ann.pl
Copying scripts/biosql/update-on-new-date.pl -> blib/script/update-on- 
new-date.pl
blib/script/update-on-new-date.pl -> blib/script/bp_update-on-new- 
date.pl
Copying scripts/biosql/terms/add-term-annot.pl -> blib/script/add-term- 
annot.pl
Deleting blib/script/add-term-annot.pl.bak
blib/script/add-term-annot.pl -> blib/script/bp_add-term-annot.pl
Copying scripts/corba/caching_corba_server.pl -> blib/script/ 
caching_corba_server.pl
Deleting blib/script/caching_corba_server.pl.bak
blib/script/caching_corba_server.pl -> blib/script/ 
bp_caching_corba_server.pl
Copying scripts/biosql/load_ontology.pl -> blib/script/load_ontology.pl
Deleting blib/script/load_ontology.pl.bak
blib/script/load_ontology.pl -> blib/script/bp_load_ontology.pl
Copying scripts/biosql/load_seqdatabase.pl -> blib/script/ 
load_seqdatabase.pl
Deleting blib/script/load_seqdatabase.pl.bak
blib/script/load_seqdatabase.pl -> blib/script/bp_load_seqdatabase.pl
Copying scripts/biosql/terms/interpro2go.pl -> blib/script/ 
interpro2go.pl
blib/script/interpro2go.pl -> blib/script/bp_interpro2go.pl
Copying scripts/biosql/clean_ontology.pl -> blib/script/ 
clean_ontology.pl
blib/script/clean_ontology.pl -> blib/script/bp_clean_ontology.pl
Copying scripts/corba/test_bioenv.pl -> blib/script/test_bioenv.pl
Deleting blib/script/test_bioenv.pl.bak
blib/script/test_bioenv.pl -> blib/script/bp_test_bioenv.pl
Copying scripts/biosql/update-on-new-version.pl -> blib/script/update- 
on-new-version.pl
blib/script/update-on-new-version.pl -> blib/script/bp_update-on-new- 
version.pl
Copying scripts/corba/bioenv_server.pl -> blib/script/bioenv_server.pl
Deleting blib/script/bioenv_server.pl.bak
blib/script/bioenv_server.pl -> blib/script/bp_bioenv_server.pl
Copying scripts/biosql/bioentry2flat.pl -> blib/script/bioentry2flat.pl
Deleting blib/script/bioentry2flat.pl.bak
blib/script/bioentry2flat.pl -> blib/script/bp_bioentry2flat.pl
Copying scripts/biosql/load_interpro.pl -> blib/script/load_interpro.pl
blib/script/load_interpro.pl -> blib/script/bp_load_interpro.pl
Copying scripts/biosql/cgi-bin/getentry.pl -> blib/script/getentry.pl
Deleting blib/script/getentry.pl.bak
blib/script/getentry.pl -> blib/script/bp_getentry.pl
Copying scripts/biosql/del-assocs-sql.pl -> blib/script/del-assocs- 
sql.pl
blib/script/del-assocs-sql.pl -> blib/script/bp_del-assocs-sql.pl
Copying scripts/biosql/freshen-annot.pl -> blib/script/freshen-annot.pl
blib/script/freshen-annot.pl -> blib/script/bp_freshen-annot.pl
t/11locuslink.t....1/113
------------- EXCEPTION: Bio::Root::Exception -------------
MSG: Unique key query in Bio::DB::BioSQL::SpeciesAdaptor returned 2  
rows instead of 1. Query was [name_class="scientific  
name",binomial="Homo sapiens"]
STACK: Error::throw
STACK: Bio::Root::Root::throw /usr/lib/perl5/site_perl/5.8.5/Bio/Root/ 
Root.pm:357
STACK: Bio::DB::BioSQL::BasePersistenceAdaptor::_find_by_unique_key / 
root/mmroot/bioperl-db/blib/lib/Bio/DB/BioSQL/ 
BasePersistenceAdaptor.pm:965
STACK: Bio::DB::BioSQL::BasePersistenceAdaptor::find_by_unique_key / 
root/mmroot/bioperl-db/blib/lib/Bio/DB/BioSQL/ 
BasePersistenceAdaptor.pm:860
STACK: Bio::DB::BioSQL::BasePersistenceAdaptor::create /root/mmroot/ 
bioperl-db/blib/lib/Bio/DB/BioSQL/BasePersistenceAdaptor.pm:182
STACK: Bio::DB::Persistent::PersistentObject::create /root/mmroot/ 
bioperl-db/blib/lib/Bio/DB/Persistent/PersistentObject.pm:244
STACK: Bio::DB::BioSQL::BasePersistenceAdaptor::create /root/mmroot/ 
bioperl-db/blib/lib/Bio/DB/BioSQL/BasePersistenceAdaptor.pm:169
STACK: Bio::DB::Persistent::PersistentObject::create /root/mmroot/ 
bioperl-db/blib/lib/Bio/DB/Persistent/PersistentObject.pm:244
STACK: t/11locuslink.t:30
-----------------------------------------------------------
# Looks like you planned 113 tests but ran 7.
# Looks like your test exited with 255 just after 7.
t/11locuslink.t.... Dubious, test returned 255 (wstat 65280, 0xff00)
  Failed 106/113 subtests

Test Summary Report
-------------------
t/11locuslink.t (Wstat: 65280 Tests: 7 Failed: 0)
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 113 tests but ran 7.
Files=1, Tests=7,  2 wallclock secs ( 0.01 usr  0.00 sys +  0.40 cusr   
0.06 csys =  0.47 CPU)
Result: FAIL
Failed 1/1 test programs. 0/7 subtests failed.

Thanks

Mike

>
>
> 	-hilmar
>
> On Feb 6, 2009, at 11:03 AM, Michael Muratet wrote:
>
>> Greetings
>>
>> I have use bioperl-db and load_seqdatabase.pl many times in the  
>> past and it's worked pretty much out of the box.
>>
>> I have been trying to load fasta files from hg18. For chr1, the  
>> virtual and resident memory quickly builds to 14 GB or so, then  
>> starts using up the 2GB swap until it's full and then the system  
>> hangs. The system is an 8-core Dell with 16GB of physical memory.  
>> chr1.fa is ~242MB. All of the disk storage is network mounted on an  
>> EMC system which (I am told) has a proprietary version of something  
>> that's NFS-like.
>>
>> I loaded chrM (~17kb) and load_seqdatabase grew to over 4 GB before  
>> it completed.
>>
>> I am using MySQL 5.0.51a-community, DBI 1.607, perl 5.85, bioperl  
>> 1.6.0, bioperl-db 1.006900.
>>
>> I have the innodb engine enabled in MySQL and the buffers and  
>> caches set for a 'large' system.
>>
>> I had some errors during the bioperl-db install:
>>
>> Test Summary Report
>> -------------------
>> t/01dbadaptor.t (Wstat: 256 Tests: 23 Failed: 1)
>> Failed test:  23
>> Non-zero exit status: 1
>> t/10ensembl.t   (Wstat: 65280 Tests: 5 Failed: 0)
>> Non-zero exit status: 255
>> Parse errors: Bad plan.  You planned 18 tests but ran 5.
>> t/11locuslink.t (Wstat: 65280 Tests: 7 Failed: 0)
>> Non-zero exit status: 255
>> Parse errors: Bad plan.  You planned 113 tests but ran 7.
>> t/15cluster.t   (Wstat: 65280 Tests: 7 Failed: 0)
>> Non-zero exit status: 255
>> Parse errors: Bad plan.  You planned 162 tests but ran 7.
>> Files=16, Tests=1205, 21 wallclock secs ( 0.51 usr  0.12 sys +  
>> 14.90 cusr  2.69 csys = 18.22 CPU)
>> Result: FAIL
>> Failed 4/16 test programs. 1/1205 subtests failed.
>>
>> The most recent bioperl-db documentation says that a workable  
>> version may be possible after some errors and so I went ahead with  
>> the install.
>>
>> The mailing list archive has some discussion about throughput but  
>> nothing really about filling up memory.
>>
>> Can anyone offer any clues about what's going on or where to start  
>> looking?
>>
>> Thanks
>>
>> Mike
>> _______________________________________________
>> Bioperl-l mailing list
>> Bioperl-l at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>
> -- 
> ===========================================================
> : Hilmar Lapp  -:-  Durham, NC  -:-  hlapp at gmx dot net :
> ===========================================================
>
>
>

Michael Muratet
mmuratet at hudsonalpha.org