[DAS2] tiled queries for performance

[Brian Gilman]

Hello Everyone,

	I've been lurking on the list and wanted to say hi.

	We're looking into this kind of implementation issue ourselves and  
thought that a bitorrent like cache makes the most sense. ie. all  
servers in the "fabric" are issued the query in a certain "hop  
adjacency". These servers then send their data to the client who's  
job it is to assemble the data.

								HTH,

										-B
--
Brian Gilman
President Panther Informatics Inc.
E-Mail: gilmanb at pantherinformatics.com
         gilmanb at jforge.net
AIM: gilmanb1

01000010 01101001 01101111
01001001 01101110 01100110
01101111 01110010 01101101
01100001 01110100 01101001
01100011 01101001 01100001
01101110



On Nov 21, 2005, at 3:47 PM, Allen Day wrote:

> Hi,
>
> I had an idea of how clients may be able to get better response from
> servers by using a tiled query technique.  Here's the basic idea:
>
> ClientA wants features in chr1/1010:2020, and issues a request for  
> that
> range.  No other clients have previously requested this range, so the
> server-side cache faults to the DAS/2 service (slow).
>
> ClientB wants features in chr1/1020:2030, and issues a request for  
> that
> range.  Although the intersection of the resulting records with  
> ClientA's
> query is large, the URIs are different and the server-side cache  
> faults
> again.
>
> If ClientA and ClientB were to each issue two separate "tiled"  
> requests:
>
>  1. chr1/1001:2000
>  2. chr1/2001:3000
>
> ClientB could take advantage of the fact that ClientA had been  
> looking at
> the same tiles.
>
> For this to work, the clients would need to be using the same tile  
> size.
> The optimal tile size is likely to vary from datasource to datasource,
> depending on the length and density distributions of the features
> contained in the datasource.  The "sources" or "versioned sources"
> payload could suggest a tiling size to prospective clients.   
> Servers could
> also pre-cache all tiles by hitting each tile after an update of the
> datasource (or the DAS/2 service code).
>
> The tradeoff for the performance gains is that clients may now need  
> to do
> filtering on the returned records to only return those requested by  
> the
> client's client.
>
> -Allen
> _______________________________________________
> DAS2 mailing list
> DAS2 at portal.open-bio.org
> http://portal.open-bio.org/mailman/listinfo/das2