[erlang-questions] how: prefix matching on external term format

[Paul Mineiro]

Oh hey since there's interest, I got this working by forking
erl_compare_ext and adding special tags for smallest and largest erlang
term.  My original fear of forking was unfounded since 1) the canonical
erl_compare_ext has several bugs and 2) the canonical erl_compare_ext does
not support ERL_NEW_FLOAT_EXT.  That last one was pretty funny since my
main fear of forking was missing out on new type tags.  Well, the
directory *does* say "legacy" ...

I'm in the end game now, within a week I should have something on google
code that sports the same interface as dets but supports ordered_set and
ordered_bag (and 8EB [9.22e18 bytes] byte file size limit).

-- p

On Wed, 4 Jun 2008, Scott Lystig Fritchie wrote:

> Paul Mineiro <paul-trapexit@REDACTED> wrote:
>
> pm> so i've envisioned the following choices:
>
> Hrm, I wonder if it's a viable option to avoid the official Erlang
> external format?  If I recall correctly(*), any tuple stored in ETS is
> not serialized using the external term format.  Instead, the tuple is
> copied into a separate heap.  That heap is big enough to store only the
> tuple.  But the data inside that heap uses the same tagged pointer
> scheme as any Erlang term inside a process's heap, so all the term
> comparison macros & functions work, regardless of type (process heap or
> ETS mini-heap).
>
> (*) If I'm wrong, someone please yell loudly at me and the list.  I
> wouldn't want the falsehood to live unrefuted forever.
>
> If my memory is correct, then storing an ETS mini-heap onto disk will
> suffer from pointer offset problems: the VM's tagged pointers for use in
> RAM won't point to the necessary byte offset within a file on disk.  The
> simplest (?) kludge would be to write an ETS mini-heap with all pointers
> rewritten to be relative to the start of the mini-heap, and then re-set
> those pointers to RAM-valid values after reading from disk but before
> any comparisons.  (Simplest = avoid reimplementing term comparison and
> match spec functions.)
>
> -Scott
>

In an artificial world, only extremists live naturally.

        -- Paul Graham