[DAS2] Dasypus status

[Andrew Dalke]

(See, I'm trying out one of the possible names)

I've implemented the core architecture over the last
few days.  It took some head-banging, and sadly not
the fun kind.

The result is somewhat sweet.  The difficulty was
because I wanted to test the various sorts of things
that could go wrong.  This include:

   - HTTP server doesn't respond, infinite loop, etc.
   - the wrong values in the HTTP headers (eg, wrong version)
   - invalid XML returned [1]
   - wrong elements in the XML [2]
   - wrong fields in the XML [3]

[1] We should add a new DAS header which is the server
version.  Eg, some of the DAS/1 servers returned invalid
XML, like <BLAH href="http://example.com/blah?a=1&b=2">
when the "&" attribute needs to be escaped.  I want
to allow a cleanup, triggered on the server type, that
fixes these sorts of errors.

[2] While the schema can catch these, some are errors
while others (like extending the XML) should be warnings.

[3] For example, the prototype server uses the wrong
namespace and wrong xml:base.  I've told Allen about
it but in the meanwhile I've put a host-based cleanup
filter in place.

In addition several layers of the architecture are
easily exchangeable with others.  For example:

   - the "URL opener" is replaceable.  This lets me
put in test code to check that the URL of a request
is as expected, lets me wrote a standalone proxy
system to use local files for http requests, and
potentially lets someone implement, say, an interface
that reads directly from a MySQL connection instead
of going through the web.  (Hmmm, interesting ... I
had just thought of that possibility.)

   - the "builder" layer is replaceable.  This is the
one that in most apps takes the input socket connection
and makes a usable data structure from it.  I have
one builder for Python data structures.  I'll soon have
one for the schema checker and another that allows
both builders to work.

   - the "build a Python data structure" knows that a
few data structures are special and could be converted
into something other than a string or integer.  One
example is the ISO date, which by default gets turned
into a Python datetime.  Another is a link from one
part of DAS to another.  By default it's a string but
I experimented with a version that fetches the given link
on request, which makes the data structure act like a
direct mirror of the DAS object network.

In doing this I noticed a couple of problems with the
DAS XML we have.  I've pointed out a couple.  I'm
in the process of cleaning up a few others.

I am able to read all of the XML in the DAS CVS
repository and Allen's SOURCES url.  The server has
been down the last few days and I haven't been able
to get any further.

The next steps for me are:
   - we were going to change the <PROP> section of the
spec, right?  What were the changes?

   - check Dasypus into CVS

   - implements builders for the RNG validation and to
let both the RNG and Python data structure builders work.

   - finish the error/warning handling code

   - clean up the example XML files

   - start developing test sets

   - work with Allen's prototype server

   - implement the handler for the compact formats
(Again, what's the reason for the compact formats?)

All that is to check that a request/response works.
The next stage is to check for system integrity.
I'm not close enough to that yet to describe the
steps involved.

					Andrew
					dalke at dalkescientific.com