[DAS] 1.6 draft 7

[Mitch Skinner]

  On 09/23/2010 07:44 AM, Thomas Down wrote:
> Just getting in touch to let you know that we're very interested in getting
> big piles of sequencing data onto genome browsers, and are certainly
> interested in alternative formats.  My current thoughts are that -- at least
> for what we're doing -- a slightly more concise XML schema, plus richer
> control of server-side summarization/binning (think maxbins++) might be the
> sweet spot, but very interested to see other options as well.

Hello,

I've only just joined the list, so I think I'm entering this 
conversation mid-way, and I apologize if I'm speaking out of turn.  I do 
have a few opinions about putting genomic data into JSON, though, and it 
looks like this might be the time to express them, if you're discussing 
DAS alternative content formats.

As you may know, JBrowse client-server communication is (currently) not 
DAS.  And that's on purpose, for multiple reasons that may be relevant 
to this discussion.  They are:

1. XML verbosity
2. cacheability
3. need for server-side code

JBrowse does something different from DAS in each of those areas.  In 
order of decreasing ease-of-adoption, they are:

1. XML verbosity:

It's true that if you gzip then this doesn't cost you too much in terms 
of size-on-the-wire, but you still have to deal with it in memory on the 
client and server.

Most of the genomics data formats I've seen have attributes that are 
either *named* or *positional*.  Positional data formats, like GFF2, 
specify in advance the attributes (columns) that each record (line) can 
have.  This saves space, but makes the format awkward to extend (e.g., 
what happened when different people extended GFF in different ways).  
With XML, by contrast, each attribute is named.  That makes the tradeoff 
the other way around; it takes up more space, but it's easier to extend, 
because new attributes get new names and don't step on each other.

And then there are hybrids like GFF3, where some attributes are 
specified positionally, and the rest are named.

But there's another middle ground--you can have positional attributes, 
but have the *positions* be named.  This is what JBrowse does; each 
feature is represented in JSON as an array.  Each position in the array 
is named, in a separate array of strings.  In other words, there are a 
bunch of genomic features like this:

[
[10000,20000,-1,"foo"],
[50000,80000,1,"bar]
]

and a separate array (not sure what to call it, a "schema" array, 
maybe?) that names each position, e.g.:

["start", "end", "strand", "name"]

or what have you.  And, since it's JSON, attribute values can have their 
own nested structure; for example, one of the attributes could be 
"subfeatures" which could be an array of subfeatures.  In JBrowse, each 
track can have its own "schema"; attributes like "phase", "score", and 
"subfeatures" can be left out of a track if it doesn't use them.

For genomic applications, where the space used by the schema array can 
be amortized over large numbers of features, I think this is a pretty 
substantial win.  And if you need ad-hoc additional attributes, you can 
always pull the GFF3 trick and have a set of named attributes at the end.

2. Cacheability

JBrowse partitions feature data by refseq, track, and genomic region.  
And the boundaries between those chunks of data are defined statically.  
That makes those chunks of data more cacheable; if you're viewing a 
given region, and you move a little to the left or to the right, you 
usually don't need to make a new HTTP request.  And if you come back to 
the same region the next day, your browser will likely have the same 
data in its cache.

You could, potentially, provide useful caching-related HTTP headers from 
current DAS servers, but my understanding is that implementations 
typically don't.  And I think that has something to do with dynamic 
queries being harder to usefully cache.

Gregg has been working on proxying DAS sources for JBrowse clients; I 
asked him if his proxy would provide useful caching headers, and he 
said, "well, I could make them up" :)  But it would be nice if we could 
get them without having to do that.

3. Server-side code

The fact that the boundaries of the data chunks are statically defined 
means that you can just pre-generate them and serve them with a plain 
static HTTP server.  Sometimes I meet people who disagree with this 
choice, but if you have have a workload that's dominated by reads then I 
think it's a pretty clear win, especially if you have a lot of users.  
Plus, the HTTP server will generate appropriate caching-related headers 
for free.  And it's easier for less-technical users, or users with 
limited rights on a server, to set up a JBrowse instance if they don't 
need to set up CGI/servlets/whatever.

BAM and BigBed have made a similar choice to be statically-serveable; 
you could view the JBrowse approach as doing something like what they're 
doing, in a way that's easier to digest for web browser clients.  In the 
JBrowse case, range queries are done by the client using a lazily-loaded 
nested containment list, described here:

http://biowiki.org/view/JBrowse/LazyFeatureLoading

It's possible to store the JSON gzipped on the server and then send it 
out as-is; I compress the json files and give them a .jsonz extension, 
and then add this to my apache config:

<Files *.jsonz>
   ForceType text/javascript
   Header set Content-Encoding: gzip
</Files>

which works in all the web browsers that JBrowse supports, including IE 
6/7/8.

I hope this contributes to the discussion,
Mitch