[MOBY-dev] data by reference - a request for comments

[Martin Senger]

I am trying to figure out how I would implement the data by reference in
order to achieve the main purpose - not no have all data in memory. I know
how I would do it for the references being on the primitive types level, but
it is less clear to me how I would do it with the references on other
levels. Perhaps you can help me to explain how you would do it (or even
Pieter can explore how he is already doing it)?

For references on the primitive type level, I would do a usual parsing of a
Moby message and when I find a reference, I would resolve it do a local file
and I would pass to my service class the data as a file reference. It would
be up to the service to read the file contents to the memory or not - that
would depend what it needs to do with the data.

However, on the higher level, such as mobyData level, the reference must be
treated differently. If I do the same (resolving the reference to a local
file), I still need to call a parser again to parse the contents of the
local file. Because I still want to give my service already parsed data (and
not a biomoby XML), and because I do not want to have all in memory, I need
to create local files for each primitive type and do the same as above
(passing my service local file references). I do not see much other choices.
Am I missing a point here?

The above is doable, of course. But you see why I wanted to have references
only on the primitive type level. If we have them on any level (as it seems
to be) I need, in my implementation, actually make another (local)
references for the primitive type, anyway. Of course, it is better than the
remote references because I do not need to make a network connection for
each individual primitive type (point taken, Pieter) - but it still to be
done.

Cheers,
Martin

-- 
Martin Senger
email: martin.senger at gmail.com,m.senger at cgiar.org
skype: martinsenger