[Bioperl-l] Re: [SO-devel] GFF3 preliminary

Lincoln Stein lstein at cshl.org
Wed Feb 19 22:43:58 EST 2003


That's what I said a few years ago, but my predictions of the death of 
tab-delimited formats just haven't come to pass.  The problem is that 
tab-delimited formats are:

	handy
	easy to read
	easy to parse
	work well with tools like grep and sort
	very space efficient

No matter how you'd like to have everyone use XML formats, they will typically 
be used for very specific applications, like the use of GAME/XML for 
drosophila genome annotation.  Even XML fans like myself fall back to using 
tab-delimited formats when push comes to shove because they're so damn easy 
and generic.

People are going to use tab-delimited formats (GFF, GTF, etc) like it or not. 
GFF3 is an attempt to give them a reasonable format that is relatively easy 
to use and makes not too many compromises.

Lincoln

On Wednesday 19 February 2003 02:25 pm, Mark Yandell wrote:
> Hi All,
>
>
> ".  When asked why they
>
> > have modified the published Sanger specification, bioinformaticists
> > frequently answer that the format was insufficient for their needs...",
>
> So why not just use XML? you know, with like a real DTD, like the rest of
> the world and be done with it ?
>
>
> --mark
>
> > Hi,
> >
> > Following up on discussions with Jim Kent, Suzi Lewis, Michele Clamp
> > and Richard Durbin, here is a new version of the GFF3 proposal.
> >
> > Suzi, could you post this to song.sourceforge.net, when you have a
> > chance?  I don't seem to have write permissions to the htdocs
> > directory.
> >
> > Best,
> >
> > Lincoln
> >
> >
> > GENERIC FEATURE FORMAT VERSION 3: A PROPOSAL
> >
> > Author:  Lincoln Stein
> > Date:    19 February 2003
> > Version: 0.2
> >
> > Although there are many richer ways of representing genomic features
> > via XML, the stubborn persistence of a variety of ad-hoc tab-delimited
> > flat file formats declares the bioinformatics community's need for a
> > simple format that can be modified with a text editor and processed
> > with shell tools like grep.  The GFF format, although widely used, has
> > fragmented into multiple incompatible dialects.  When asked why they
> > have modified the published Sanger specification, bioinformaticists
> > frequently answer that the format was insufficient for their needs,
> > and they needed to extend it.  The proposed GFF3 format addresses the
> > most common extensions to GFF, while preserving backward compatibility
> > with previous formats. The new format:
> >
> >     1) adds a mechanism for representing more than one level
> >        of hierarchical grouping of features and subfeatures.
> >     2) separates the ideas of group membership and feature name/id
> >     3) constrains the feature type field to be taken from a controlled
> >        vocabulary.
> >     4) allows a single feature, such as an exon, to belong to more than
> >        one group at a time.
> >     5) one level of relative addressing for subfeatures (e.g. exons
> >        can be expressed in transcript coordinates)
> >     6) an explicit convention for pairwise alignments
> >     7) an explicit convention for features that occupy disjunct regions
> >
> > The format consists of 10 columns, separated by spaces.  The following
> > unescaped characters are allowed within fields:
> > [a-zA-Z0-9.:;=%^*$@!+_?-].  All other characters must must be escaped
> > using the URL escaping conventions.  Unescaped quotation marks,
> > backslashes and other ad-hoc escaping conventions that have been added
> > to the GFF format are explicitly forbidden.  The =, ; and % characters
> > have reserved meanings as described below.
> >
> > Undefined fields are replaced with the "." character, as described in
> > the original GFF spec.
> >
> > Column 1: "seqid"
> >
> > The ID of the landmark used to establish the coordinate system for the
> > current feature.  IDs must contain alphanumeric characters.
> > Whitespace, if present, must be escaped using URL escaping rules
> > (e.g. space="%20" or "+").
> >
> > Column 2: "source"
> >
> > The source of the feature.  This is unchanged from the older GFF specs
> > and is not part of a controlled vocabulary.
> >
> > Column 3: "type"
> >
> > The type of the feature (previously called the "method").  This is
> > constrained to be either: (a) a term from SOFA; or (b) a SOFA
> > accession number.  The latter alternative is distinguished using the
> > syntax SOFA:000000.
> >
> > Columns 4 & 5: "start" and "end"
> >
> > The start and end of the feature, in 1-based integer coordinates,
> > relative to the landmark given in column 1.  Start is less than end.
> >
> > Column 6: "score"
> >
> > The score of the feature, a floating point number.  As in earlier
> > versions of the format, the semantics of the score are ill-defined.
> > It is strongly recommended that E-values be used for sequence
> > similarity features, and that P-values be used for ab initio gene
> > prediction features.
> >
> > Column 7: "strand"
> >
> > The strand of the feature.  + for positive strand (relative to the
> > landmark), - for minus strand, and . for features that are not
> > stranded.  In addition, ? can be used for features whose strandedness
> > is relevant, but unknown.
> >
> > Column 8: "phase"
> >
> > The phase of the feature, for protein-encoding featues (primarily
> > CDSs).  This is an integer-valued field with the values 0, 1, or 2.
> > The integer indicates the offset from the start of the feature to the
> > first base of the first codon in the reading frame.  "." is used for
> > features that do not corresponding to a reading frame.
> >
> > Column 9: "attributes"
> >
> > A list of feature attributes in the format tag=value.  Multiple
> > tag=value pairs are separated by semicolons.  URL escaping rules are
> > used for tags or values containing the following characters: ",=;".
> > Whitespace should be replaced with the "+" character or the %20 URL
> > escape.  This will allow the file to survive text processing programs
> > that convert tabs into spaces.
> >
> > Five tags are predefined:
> >
> >     ID	   Indicates the name of the feature.  IDs must be unique
> > 	   within the scope of the GFF file.
> >
> >     Alias  A descriptive name for the feature.  It is suggested that
> > 	   this tag be used whenever a secondary identifier for the
> > 	   feature is needed, such as display names, locus names and
> > 	   accession numbers.  Unlike ID, there is no requirement
> > 	   that Alias be unique within the file.
> >
> >     Parent Indicates the parent of the feature.  A parent ID can be
> > 	   used to group exons into transcripts, transcripts into
> > 	   genes, an so forth.  A feature may have multiple parents.
> >
> >     Target Indicates the target of a nucleotide to nucleotide or
> > 	   nucleotide to protein alignment.  The format of the
> > 	   value is "target_id:start..end"  Start may be greater
> > 	   than end to indicate a + strand alignment to the
> > 	   reverse complement of a target nucleotide sequence.
> >
> >     Align  The alignment of the feature to the target if the two
> > 	   are not colinear.  The alignment is a string containing
> > 	   the four characters "|X^v", where "|" indicates an
> > 	   aligned match, "X" indicates an aligned mismatch, "^"
> > 	   indicates a gap in the feature, and "v" indicates a
> > 	   gap in the target.
> >
> > Multiple attributes of the same type are indicated by separating the
> > values with the comma "," character, as in:
> >
> >        Parent=AF2312,AB2812,abc-3
> >
> > Note that attribute names are case sensitive.  "Parent" is not the
> > same as "parent".
> >
> > In the example GFF3 file given below, the first column contains line
> > numbers that I have added for the purposes of the narrative.  Here are
> > some common scenarios that I have attempted to illustrate:
> >
> > A) a simple feature, no public ID
> >
> > Line 2 in the example is a feature of type "repeat". It is located on
> > the coordinate system defined by feature "ctg123", has a start and an
> > end and no ID.  It has an attribute named "Note" with value "ALU3."
> >
> > B) a simple feature with a public ID
> >
> > Line 3 is a feature of type clone.  It has a start and an end.  Its
> > parent is undefined (no Parent attribute), but it has an ID attribute
> > of "clone00001" and an Alias of "cTel33B."
> >
> > C) a feature with multiple attributes
> >
> > Line 5 is a feature of type "gene."  It has no parent, and has
> > attributes of type ID, Note, and GO_term.
> >
> > D) a hierarchical grouping of features
> >
> > Lines 5-13 demonstrate a hierarchical grouping.  At the top level is
> > line 5, which defines the extent of a "gene" with ID "gene00001".
> > Below this are two features of type mRNA (lines 6 and 7).  Their
> > Parent attributes are set to "gene00001", indicating that this feature
> > is their immediate parent.  Their IDs are indicated as separate
> > attributes.
> >
> > This pattern is repeated for the exons listed on lines 8-11.  Exons
> > exon00001, exon00002, and exon00004 belong to both of the transcripts.
> > Therefore, their Parent attribute contains both the mRNA00001 and
> > mRNA00002 IDs separated by a comma.
> >
> > Exon exon00003 belongs to mRNA00002 only, and therefore that
> > transcript's ID is listed as the sole Parent.
> >
> > Lines 12 and 13 indicate coding_start and coding_end features.  These
> > subfeatures are hierarchically grouped underneath their corresponding
> > exons, but they do not have independent public IDs.
> >
> > E) Disjunct coordinates
> >
> > Lines 14-16 illustrates a single feature -- the CDS corresponding to
> > mRNA mRNA00001 -- which occupies multiple disjunct regions.  The
> > Parent attribute indicates that the CDS features belong to mRNA00001.
> > However, the attribute column assigns each of lines 14-16 the same ID.
> > Because the ID is the same, this is interpreted as a single feature
> > that spans multiple disjunct coordinate ranges.
> >
> > NOTE: See "Representing Translations" for a discussion of why it might
> > not be a good idea to use represent translations in this way.
> >
> > F) Alignments
> >
> > Lines 17-19 demonstrate an alignment of two sequences using the
> > reserved Target attribute.  Each non-gapped segment becomes a line in
> > the GFF3 file.  The segments each share the same ID, thereby
> > indicating that the segments are disjunct regions of the same feature.
> > The Target attribute indicates the ID of the target sequence (which
> > does not have to be represented in the GFF3 file) and the start and
> > end coordinates of the aligned target.
> >
> > Line 20 shows a gapped alignment using the Align attribute.  This
> > attribute's value should be interpreted this way:
> >
> >
> >  1501  gatt*ctccc 1510      ctg123
> >
> >        ||||^||X||
> >
> >  2001  gatttctgcc 2011      af923
> >
> > Unlike the GFF1 and GFF2 formats, the Parent attribute for gapped
> > alignments may be empty. However, a valid alternative representation
> > is to create a single "match" feature, and a series of "hsp" features
> > contained within it.  Lines 21-23 show this alternative
> > representation.
> >
> > G) Relative coordinates
> >
> > Lines 24-27 illustrate using relative coordinate addressing in
> > feature/subfeature relationships.  Line 24 defines an mRNA that is
> > positioned on sequence landmark "ctg123" from positions 5000 to 6000.
> > Its ID field indicates that is mRNA03.  Lines 25-27 are exon
> > subfeatures of mRNA03 as indicated by their Parent attribute.
> > However, the seqid field specifies mRNA03 as the parent coordinate
> > system, thereby allowing the exons to begin at position 1.
> >
> >   0  ##gff-version 3
> >   1  ##sequence-region ctg123:1..1497228
> >
> >   2  ctg123  flybase repeat  5000    5100    .       .       .
>
> Note=ALU3
>
> >   3  ctg123  flybase clone   1       2679    .       +       .
>
> ID=clone00001;Alias=cTel33B
>
> >   4  ctg123  flybase contig  1       1497228 .       +       .
>
> ID=contig0001;Alias=ctg123
>
> >   5  ctg123  flybase gene    43733   44677   .       +       .
>
> ID=gene00001;Alias=ADAM1;Note=unc-3;GO_term=GO:12345,GO:33421
>
> >   6  ctg123  flybase mRNA    43733   44677   .       +       .
>
> ID=mRNA00001;Alias=ADAM1.t1;Parent=gene00001
>
> >   7  ctg123  flybase mRNA    43733   44677   .       +       .
>
> ID=mRNA00002;Alias=ADAM1.t2;Parent=gene00001
>
> >   8  ctg123  flybase exon    43733   43961   .       +       .
>
> ID=exon00001;Parent=mRNA00001,mRNA00002
>
> >   9  ctg123  flybase exon    44030   44234   .       +       .
>
> ID=exon00002;Parent=mRNA00001,mRNA00002
>
> >  10  ctg123  flybase exon    44281   44328   .       +       .
>
> ID=exon00003;Parant=mRNA00002
>
> >  11  ctg123  flybase exon    44521   44677   .       +       .
>
> ID=exon00004;Parent=mRNA00001,mRNA00002
>
> >  12  ctg123  flybase coding_start    43740   43740   .       +       .
>
> Parent=exon00001
>
> >  13  ctg123  flybase coding_end      44677   44677   .       +       .
>
> Parent=exon00004
>
> >  14  ctg123  flybase cds     43740   43961   .       +       0
>
> ID=cds00001;Parent=mRNA00001
>
> >  15  ctg123  flybase cds     44030   44234   .       +       1
>
> ID=cds00001;Parent=mRNA00001
>
> >  16  ctg123  flybase cds     44521   44677   .       +       1
>
> ID=cds00001;Parent=mRNA00001
>
> >  17  ctg123  flybase match   1       100     100     .       .
>
> ID=match0001;Target=af923:1001..1100
>
> >  18  ctg123  flybase match   101     500     80      .       .
>
> ID=match0001;Target=af923:1101..1500
>
> >  19  ctg123  flybase match   501     1000    80      .       .
>
> ID=match0001;Target=af923:1501..2000
>
> >  20  ctg123  flybase match   1501    1510    60      .       .
>
> ID=match0001;Target=af923:2001..2011;Align=||||^||X||
>
> >  21  ctg123  flybase match   5001    6000    100     .       .
>
> ID=match0002;Target=ua388:1..1000
>
> >  22  ctg123  flybase hsp     5001    5500    .       .       .
>
> Parent=match0002;Target=ua388:1..500
>
> >  23  ctg123  flybase hsp     5501    6000    .       .       .
>
> Parent=match0002;Target-ua388:501.1000
>
> >  24  ctg123  flybase mRNA    5000    6000    +       .       .
>
> ID=mRNA03;Alias=EVE1.t1
>
> >  25  mRNA03  flybase exon    1       300     +       .       .
>
> ID=exon00005;Parent=mRNA03
>
> >  26  mRNA03  flybase exon    301     400     +       .       .
>
> ID=exon00006;Parent=mRNA03
>
> >  27  mRNA03  flybase exon    401     1000    +       .       .
>
> ID=exon00007;Parent=mRNA03
>
> > =================================================================
> >
> > OTHER SYNTAX:
> >
> > Comments are preceded by the # symbol.  Meta-data and directives are
> > preceded by ##.  The following directives are recognized:
> >
> >   ##gff-version 3
> > 	The GFF version, always 3 in this spec.  This must
> > 	be the topmost line of the file.
> >
> >   ##sequence-region seqid:start..end
> >         The sequence segment referred to
> > 	by this file, in the format seqid:start..end.
> > 	This element is optional.  If it occurs, it must be
> > 	the second line of the file.
> >
> >   ###
> >         This directive (three # signs in a row) indicates that all
> >         forward references to feature IDs that have been seen to this
> >         point have been resolved.  After seeing this directive, a
> >         program that is processing the file serially can close off any
> >         open objects that it has created and return them, thereby
> >         allowing iterative access to the file.  Otherwise, software
> >         cannot know that a feature has been fully populated by its
> >         subfeatures until the end of the file has been reached.
> >
> >
> > =================================================================
> >
> > REPRESENTING TRANSLATIONS
> >
> > There are two ways of representing protein translations (e.g. ORFS,
> > CDS) in the various implementations of GFF2 and GTF.  One way is to
> > represent the translation as an interrupted "CDS" region beginning
> > with the first base of the first codon and ending at the last base of
> > the stop codon.  Another is to create a series of exons and to
> > indicate the position of the translational start and end on the first
> > and last coding exon.
> >
> > An informal sampling of members of this list (Michele Clamp, Suzi
> > Lewis, Richard Durbin) suggests that the latter solution is cleaner
> > and more manageable in practice, leading to more consistent annotation
> > and to fewer ambiguities.  Therefore, I would propose that we
> > legislate that translations be represented implicitly by explicit
> > translational start and end positions.  For this to work properly, the
> > parent of the start and end sites must be the mRNA feature and NOT the
> > exon.
> >
> > Under this model, here is a generic gene
> >
> >   gene:  a bag of features, including regulatory motifs
> >      mRNA
> > 	exon
> > 	coding_start
> > 	coding_end
> > 	splice_donor
> > 	splice_acceptor
> > 	5_utr
> > 	3_utr
> >
> > Importantly, the UTRs, coding start and coding end are all children of
> > the mRNA.  Making them children of the exon (which some will be
> > tempted to do!) creates ambiguities in the interpretation of
> > alternative splices.
> >
> > =================================================================
> >
> > EXAMPLE PROGRAM
> >
> > I have extended (in an experimental way), the Bio::Tools::GFF module
> > to accomodate this new format.  Here is a test script and its output
> > when run on the above file.
> >
> >   0  #!/usr/bin/perl -w
> >   1  use strict;
> >   2  use lib '.';
> >
> >   3  use Bio::Tools::GFF;
> >   4  my $file = 'gff3.txt';
> >   5  my $gffio = Bio::Tools::GFF->new(-file=>$file,-gff_version=>3);
> >   6  my @f = sort {$a->primary_tag cmp $b->primary_tag} $gffio->features;
> >   7  format_features(\@f);
> >
> >   8  sub format_features {
> >   9    my $features = shift;
> >  10    my $tabs     = shift || 0;
> >  11    for my $f (@$features) {
> >  12      my $type  = $f->primary_tag;
> >  13      my $id    = $f->unique_id;
> >  14      $id       ||= '(no id)';# if $id =~ /HASH/;
> >  15      my ($start,$end) = ($f->start,$f->end);
> >  16      my $hit = $f->can('hstart') ?
>
> $f->hunique_id.":".$f->feature2->location->to_FTstring
>
> >  17                                  : '';
> >  18      print
>
> "\t"x$tabs,join("\t",$id,$type,$f->location->to_FTstring,$hit),"\n";
>
> >  19      format_features([$f->sub_SeqFeature],$tabs+1);
> >  20    }
> >  21  }
> >
> > OUTPUT:
> >
> > clone00001	clone	1..2679
> > contig0001	contig	1..1497228
> > gene00001	gene	43733..44677
> > 	mRNA00001	mRNA	43733..44677
> > 		exon00001	exon	43733..43961
> > 			(no id)	coding_start	43740
> > 		exon00002	exon	44030..44234
> > 		exon00004	exon	44521..44677
> > 			(no id)	coding_end	44677
> > 		cds00001	cds	join(43740..43961,44030..44234,44521..44677)
> > 	mRNA00002	mRNA	43733..44677
> > 		exon00001	exon	43733..43961
> > 			(no id)	coding_start	43740
> > 		exon00002	exon	44030..44234
> > 		exon00003	exon	44281..44328
> > 		exon00004	exon	44521..44677
> > 			(no id)	coding_end	44677
> > mRNA03	mRNA	5000..6000
> > 	exon00005	exon	5000..5299
> > 	exon00006	exon	5300..5399
> > 	exon00007	exon	5400..5999
> > match0001	match	join(1..100,101..500,501..1000,1501..1510)
>
> af923:join(1001..1100,1101..1500,1501..2000,2001..2011)
>
> > match0002	match	5001..6000	ua388:1..1000
> > 	(no id)	hsp	5001..5500	ua388:1..500
> > 	(no id)	hsp	5501..6000	ua388:501..1000
> > (no id)	repeat	5000..5100
> >
> >
> >
> > --
> > ========================================================================
> > Lincoln D. Stein                           Cold Spring Harbor Laboratory
> > lstein at cshl.org			                  Cold Spring Harbor, NY
> > 	1 Bungtown Road, Cold Spring Harbor, NY 11724
> > ========================================================================
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: SlickEdit Inc. Develop an edge.
> > The most comprehensive and flexible code editor you can use.
> > Code faster. C/C++, C#, Java, HTML, XML, many more. FREE 30-Day Trial.
> > www.slickedit.com/sourceforge
> > _______________________________________________
> > SOng-devel mailing list
> > SOng-devel at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/song-devel

-- 
========================================================================
Lincoln D. Stein                           Cold Spring Harbor Laboratory
lstein at cshl.org			                  Cold Spring Harbor, NY
	1 Bungtown Road, Cold Spring Harbor, NY 11724
========================================================================



More information about the Bioperl-l mailing list