Untimely garbage collection

[Shawn Pearce]

I'm having a little trouble abusing Erts.  :-)

What I've setup is an in-process C driver on Linux which
allocates a pool of ErlDrvBinary objects when the port
is opened from Erts.  As this driver receives data from
the bt848 frame grabber card (its a video capture driver),
it places the video data into the available binaries and
then sends the binaries to Erts with driver_output_term.

Within Erts, a pair of gen_server processes open two
ports:  one to the bt848 video driver, and another to
an X11 XVideo display driver.  The messages sent by
the bt848 with binaries attached is accepted by the
one gen_server and is directly forwarded to the other.

When the second gen_server gets the binaries, it sends
them to the port using Port ! {self(), {command, List}},
where List is the List of ErlDrvBinary objects given to
Erts by the bt848 driver.

So we have a traffic flow like this:

  bt848 ---> gen_server 1 --> gen_server 2 --> XVideo

Initial testing showed that allocating ErlDrvBinary
objects for each video frame was far too costly in
CPU time.  The allocator is just too slow.  So my
initial design was to have the bt848 driver use a
circular queue and just overwrite binaries as it
wraps around.  This causes a nice little side effect
of binaries being modified within Erlang when they
should be read only.

So I decided to use the ErlDrvBinary refc field.
If refc is 1, the bt848 driver owns the binary and
is free to modify the contents.  Erlang doesn't have
a reference, so its not a problem.  If refc is > 1,
then at least one or more processes within Erlang
still hold a reference to this binary and it cannot
be updated.  The bt848 driver uses its own micro
GC routine which just scans the queue for any
binary objects with refc == 1.

This scheme worked well, provided the gen_server's
used something like:

	handle_info({_, read, List}, State) ->
		% work with binaries stored in List
		force_gc({noreply, State}).

	force_gc(Ret) ->
		garbage_collect(),
		Ret.

This seemed kind of risky, but appeared to work well.
Erts was making the tail call into force_gc, which
allowed it to remove List from the process stack.
Since the only references to the binaries were held
in List, and its now popped off the stack, the
binaries should be unreachable and have their
refc decremented when the garbage collection occurs.
Additionally, the gen_server's use a memory heap of only
377 words, which should GC quickly.

Suddenly this has stopped working.  My C drivers are
seizing when they run out of binaries, and all of the
binaries have a refc of 3:  one for the driver that
"owns" the binary, and one for each gen_server process.

The XVideo driver only holds the binary for a short
period of time and is definately increasing and
decreasing refc properly.
refc

>From the perspective of my application, it would be ok
for my Erlang servers to notify the C drivers when they
are done with the binary so it can rewrite it, regardless
of the refc.  It just causes a lot more coordination code
to be written and more messages to be passed per set of
frames being moved.  It also breaks the Erlang single
assignment / everything is read only model, making it
harder for other modules to integrate well.

Does anyone who knows more about the Erlang GC and
Erlang driver development have better suggestions
than what I currently have?

I'm starting to get a little frustrated debugging
this "lack of GC" sorta-deadlock I'm in...

--
Shawn.

Why do I like Perl?  Because ``in accordance with Unix tradition Perl
gives you enough rope to hang yourself with.''

Why do I dislike Java? Because ``the class ROPE that should contain the
method HANG to do the hanging doesn't exist because there is too much
'security' built into the base language.''